Handheld product six-side completion state determination method based on cross-view image fusion

Abstract

The application provides a hand-held product six-surface completion state judgment method based on cross-view image fusion, relates to the technical field of data processing, and comprises the following steps: calculating a line-of-sight vector of an O point pointing to a pupil center based on pupil coordinates and spatial consistency coordinates; calculating product body axial vectors of the O point pointing to each reference point based on X, Y and Z point coordinates; calculating azimuth and pitch angle characteristic quantities based on the line-of-sight vector and the axial vector, segmenting quadrants through a spatial cutting algorithm, and obtaining a detected surface result in combination with a preset spatial azimuth mapping rule; activating a corresponding accumulator based on the detected surface result, accumulating a continuous detection duration, and comparing the continuous detection duration with a preset threshold value; if the threshold value is reached, a completion signal is generated, otherwise, an audible and light alarm is triggered. The application improves the detection reliability and result reliability in a complex operation scene.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for determining the six-sided completion status of a handheld product based on cross-view image fusion. Background Technology

[0002] In the manufacturing sectors of consumer electronics and precision instruments, the inspection of the six-sided appearance quality and assembly integrity of handheld products (such as smartphones, smartwatches, and portable sensors) is a crucial step in ensuring product reliability. Taking smartphones as an example, the inspection of six features, including the fit of the front and back covers, the tightness of the frame screws, and the integrity of the interfaces, directly impacts user experience and brand reputation. Traditional inspection methods sometimes rely on manual handheld inspection of each product under a fixed light source. This process is time-consuming for a single device and is susceptible to operator fatigue and experience differences, resulting in a high rate of missed defects.

[0003] Existing methods sometimes use independent calibration to obtain the intrinsic parameters of each camera without establishing a unified coordinate system, resulting in systematic deviations in the pose data of different cameras. For example, there are significant errors in coordinate transformation between cameras. When the product moves, the error in 3D pose calculation accumulates with distance, ultimately leading to incorrect identification of the inspected surface. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for determining the six-sided completion status of handheld products based on cross-view image fusion, which improves the detection reliability and result credibility in complex operation scenarios.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] Firstly, a method for determining the six-sided completion status of a handheld product based on cross-view image fusion, the method comprising:

[0007] Step 1: Establish a global world coordinate system A and calibrate the three-dimensional coordinates of the operator's pupil center; establish a product body coordinate system B and calibrate the three-dimensional coordinates of the origin O and the orthogonal axial reference points X, Y, and Z.

[0008] Step 2: Based on the reference point coordinates, drive the non-common-view multi-camera array to synchronously capture the image of the surface to be inspected. After the distortion is corrected by inverse projection transformation, the sub-pixel level two-dimensional coordinates of the key points on the imaging plane are calculated by a convolutional neural network.

[0009] Step 3: Based on the sub-pixel 2D coordinates and the spatial coordinates of coordinate system B, the PnP algorithm is used to iteratively calculate the 3D pose data of point O and points X, Y, and Z in each camera coordinate system.

[0010] Step 4: Based on the 3D pose data, apply a pre-calibrated rigid transformation matrix to map to coordinate system A, and use a spatial consistency fusion strategy to generate the 3D spatially consistent coordinates of point O and points X, Y, and Z in coordinate system A.

[0011] Step 5: Based on the pupil coordinates and spatial consistency coordinates, calculate the line-of-sight vector from point O to the center of the pupil; based on the X, Y, and Z coordinates, calculate the product body axial vector from point O to each reference point.

[0012] Step 6: Based on the line-of-sight vector and axis vector, calculate the azimuth and pitch angle features, divide the quadrants using a spatial cutting algorithm, and obtain the test surface result by combining the preset spatial orientation mapping rules.

[0013] Step 7: Activate the corresponding accumulator based on the inspection results, accumulate the continuous inspection time and compare it with the preset threshold. If the threshold is reached, a completion signal is generated; otherwise, an audible and visual alarm is triggered.

[0014] Furthermore, step 2 includes:

[0015] Step 21: Generate a spatial trigger signal based on the spatial coordinates of the reference point to drive the multi-camera array with non-common view layout to perform hardware-level synchronous exposure and generate a set of original images of the surface to be inspected with spatiotemporal alignment.

[0016] Step 22: Load the pre-calibrated camera intrinsic parameters and distortion parameters onto the original image set, map the pixel coordinates to the normalized plane through inverse projection transformation, and use a nonlinear compensation function to eliminate radial and tangential distortion to obtain a geometrically corrected image set.

[0017] Step 23: Define the feature search region based on the spatial coordinates of the reference point, perform multi-scale feature extraction on the geometrically corrected image set, and generate the pixel-level initial coordinates of the key points; using the pixel-level initial coordinates as the reference, calculate the gray-level gradient center offset vector in the local neighborhood, and calculate the sub-pixel-level two-dimensional coordinates of the key points.

[0018] Step 24: Establish a multi-view projection mapping relationship between the sub-pixel level two-dimensional coordinates and the reference point spatial coordinates, and calculate the reprojection error value; if the reprojection error value exceeds the tolerance threshold, adjust the feature search region parameters and re-execute; otherwise, generate the sub-pixel coordinate dataset.

[0019] Furthermore, based on sub-pixel 2D coordinates and spatial coordinates of coordinate system B, the PnP algorithm iteratively calculates the 3D pose data of point O and points X, Y, and Z in each camera coordinate system, including:

[0020] Step 31: Establish a projection correspondence set between two-dimensional imaging coordinates and three-dimensional spatial coordinates based on the sub-pixel coordinate dataset and the calibrated coordinate system B spatial coordinates; and parse the weighted representation relationship of spatial reference points based on the three-dimensional spatial point coordinates through a virtual control point mechanism.

[0021] Step 32: Based on the coordinates of the two-dimensional imaging points and the weighted representation relationship, construct the camera projection constraint equation; solve the projection constraint equation by matrix decomposition to obtain the coordinates of the virtual control points in the camera coordinate system; reconstruct the initial position of the spatial reference point in the camera coordinate system according to the coordinates of the virtual control points and the weighted representation relationship.

[0022] Step 33: Based on the initial position of the spatial reference point, calculate the pose parameters from the product coordinate system to the camera coordinate system; perform iterative optimization of the reprojection error value based on the initial position and pose parameters until convergence, and obtain the optimized 3D pose data of the camera coordinate system.

[0023] Furthermore, based on the 3D pose data, a pre-calibrated rigid transformation matrix is ​​applied and mapped to coordinate system A. A spatial consistency fusion strategy is used to generate 3D spatially consistent coordinates of point O and points X, Y, and Z in coordinate system A, including:

[0024] Step 41: Based on the 3D pose data, convert it into initial coordinates in global coordinate system A through a pre-calibrated rigid transformation matrix; based on the initial coordinates in coordinate system A and the time difference between camera exposure, calculate the motion offset and correct the coordinate position to generate a time-synchronized coordinate set;

[0025] Step 42: Calculate the fusion weight of each camera's observation data based on the reprojection error value of the pose calculation and the fluctuation range of the time-synchronized coordinate set; and generate the reference point fusion coordinates by performing a weighted average calculation based on the time-synchronized coordinate set and the fusion weight.

[0026] Step 43: Calculate the spacing between adjacent reference points in the fused coordinates, compare it with the original spacing of the calibrated coordinate system B, eliminate the spacing deviation through position offset iterative adjustment, and output the coordinates that meet the spacing consistency as the three-dimensional space consistent coordinates.

[0027] Furthermore, based on pupil coordinates and spatial consistency coordinates, the gaze vector from point O to the pupil center is calculated; based on the X, Y, and Z coordinates, the product body axial vector from point O to each reference point is calculated, including:

[0028] Step 51: Obtain the three-dimensional spatial coordinates of the operator's pupil center in the global world coordinate system A, and obtain the three-dimensional spatial consistency coordinates of the product body coordinate system origin O and the reference points X, Y, Z in the global world coordinate system A.

[0029] Step 52: In the global world coordinate system A, perform spatial vector calculation based on the coordinates of the origin O and the pupil center to generate a three-dimensional gaze vector pointing from point O to the pupil center.

[0030] Step 53: In the global world coordinate system A, perform spatial vector calculations based on the coordinates of the origin O and the coordinates of the reference points X, Y, and Z, respectively, and generate the X-axis vector pointing from point O to reference point X, the Y-axis vector pointing to reference point Y, and the Z-axis vector pointing to reference point Z.

[0031] Furthermore, based on the line-of-sight vector and axis vector, azimuth and elevation angle features are calculated. Quadrants are segmented using a spatial cutting algorithm, and the inspected surface result is obtained by combining it with a preset spatial orientation mapping rule, including:

[0032] Step 61: Calculate the projection vector on the horizontal plane of the global world coordinate system A based on the line of sight vector. Combine the projection directions of the X-axis vector and Y-axis vector of the product body coordinate system B on the horizontal plane to dynamically generate the reference azimuth reference axis. Calculate the azimuth feature between the projection vector and the reference azimuth reference axis.

[0033] Step 62: Calculate the angle between the line-of-sight vector and the horizontal projection vector to generate the pitch angle feature; perform azimuth quadrant segmentation based on the azimuth feature and pitch interval segmentation based on the pitch angle feature to form a combined spatial quadrant region.

[0034] In step 63, the combined spatial quadrant region is input into a preset spatial orientation mapping rule. The mapping rule associates the spatial quadrant with the correspondence between the product's hexahedral surface to obtain the test surface result.

[0035] Furthermore, based on the results of the inspected surface, the corresponding accumulator is activated to accumulate the continuous inspection time and compare it with a preset threshold. If the threshold is reached, a completion signal is generated; otherwise, an audible and visual alarm is triggered, including:

[0036] Step 71: Based on the results of the inspected surface, retrieve the current value of the continuous inspection duration accumulator corresponding to the inspected surface; add the current value of the accumulator to the time step value of the current detection cycle to generate the updated accumulated duration;

[0037] Determine whether the updated cumulative duration reaches the preset minimum detection duration threshold;

[0038] Step 72: If the threshold is reached, a completion signal for the inspected surface is generated and the accumulator is locked; if the threshold is not reached, the accumulator update value is stored as the input reference for the next detection cycle; when a change in the inspected surface result is detected, the original accumulator update value for the inspected surface is retrieved; if the update value is less than the minimum detection time threshold, an audible and visual alarm device is triggered and the accumulator is reset; if the threshold has been reached, the completion status is maintained and no alarm is triggered.

[0039] Furthermore, pre-calibrated camera intrinsic parameters and distortion parameters are loaded onto the original image set. The pixel coordinates are mapped to a normalized plane through inverse projection transformation. A nonlinear compensation function is used to eliminate radial and tangential distortions, resulting in a geometrically corrected image set, including:

[0040] Step 81: Obtain the two-dimensional coordinates of each pixel in the original image set and load the pre-calibrated camera intrinsic parameter matrix; construct the inverse projection mapping relationship from the pixel coordinate system to the normalized imaging plane based on the camera intrinsic parameter matrix; apply the inverse projection mapping relationship to perform coordinate transformation calculation on each pixel to generate the corresponding normalized plane coordinates; aggregate the normalized plane coordinates of all pixels to generate an uncorrected normalized coordinate set.

[0041] Step 82: Based on the uncorrected normalized coordinate set, load the pre-calibrated radial distortion parameters and tangential distortion parameters, apply the nonlinear compensation function to calculate the distortion offset, and generate the corrected normalized coordinate set.

[0042] Step 83: Based on the corrected normalized coordinate set, reconstruct the geometric position in the pixel coordinate system through the inverse mapping relationship to generate a geometrically corrected image set.

[0043] Furthermore, based on the uncorrected normalized coordinate set, pre-calibrated radial and tangential distortion parameters are loaded, and a nonlinear compensation function is applied to calculate the distortion offset, generating a corrected normalized coordinate set, including:

[0044] Step 91: Load the pre-calibrated radial distortion parameter sequence and tangential distortion parameter sequence, and obtain the uncorrected normalized coordinate set; for each coordinate point in the normalized coordinate set, calculate the squared radial distance from it to the origin of the normalized plane; based on the radial distortion parameter sequence and the squared radial distance, calculate the radial distortion scaling factor.

[0045] Step 92: Multiply the radial distortion scaling factor with the original coordinate point coordinate value to generate the radial distortion offset component; calculate the coordinate product term and the radius square term based on the tangential distortion parameter sequence and the original coordinate point coordinate value; sum the coordinate product term and the radius square term according to the tangential distortion parameter to generate the tangential distortion offset component.

[0046] Step 93: The radial distortion offset component and the tangential distortion offset component are vector-superimposed to generate the total distortion offset; the total distortion offset is subtracted from the original uncorrected coordinates to generate the corrected normalized coordinates; all corrected coordinate points are aggregated to generate the corrected normalized coordinate set.

[0047] The above-described solution of the present invention has at least the following beneficial effects:

[0048] This invention employs an inverse projection transformation algorithm to correct imaging distortion, then uses a convolutional neural network model to calculate the sub-pixel-level two-dimensional coordinates of key points, and combines this with the PnP algorithm to iteratively optimize the three-dimensional pose data. Compared to traditional pose calculation methods based on pixel-level features, sub-pixel-level precision feature extraction effectively reduces the interference of image noise on the positioning results, reduces the pose calculation error of the product's axial reference points, and provides high-quality input for accurate calculation of spatial azimuth and pitch angles, thereby reducing the risk of misjudgment of the inspected surface from the source.

[0049] By employing a dual determination mechanism of a continuous inspection duration accumulator and a minimum detection duration threshold, invalid determinations caused by instantaneous image interference or operational jitter are avoided. Compared to existing technologies that rely solely on image sharpness for determination, this strategy comprehensively considers the spatiotemporal stability of the inspected surface. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the method for determining the six-sided completion status of a handheld product based on cross-view image fusion, as provided in an embodiment of the present invention. Detailed Implementation

[0051] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0052] like Figure 1 As shown, embodiments of the present invention propose a method for determining the six-sided completion status of a handheld product based on cross-view image fusion. The method includes the following steps:

[0053] Step 1: Establish a global world coordinate system A and calibrate the three-dimensional coordinates of the operator's pupil center; establish a product body coordinate system B and calibrate the three-dimensional coordinates of the origin O and the orthogonal axial reference points X, Y, and Z.

[0054] Step 2: Based on the reference point coordinates, drive the non-common-view multi-camera array to synchronously capture the image of the surface to be inspected. After the distortion is corrected by inverse projection transformation, the sub-pixel level two-dimensional coordinates of the key points on the imaging plane are calculated by a convolutional neural network.

[0055] Step 3: Based on the sub-pixel 2D coordinates and the spatial coordinates of coordinate system B, the PnP algorithm is used to iteratively calculate the 3D pose data of point O and points X, Y, and Z in each camera coordinate system.

[0056] Step 4: Based on the 3D pose data, apply a pre-calibrated rigid transformation matrix to map to coordinate system A, and use a spatial consistency fusion strategy to generate the 3D spatially consistent coordinates of point O and points X, Y, and Z in coordinate system A.

[0057] Step 5: Based on the pupil coordinates and spatial consistency coordinates, calculate the line-of-sight vector from point O to the center of the pupil; based on the X, Y, and Z coordinates, calculate the product body axial vector from point O to each reference point.

[0058] Step 6: Based on the line-of-sight vector and axis vector, calculate the azimuth and pitch angle features, divide the quadrants using a spatial cutting algorithm, and obtain the test surface result by combining the preset spatial orientation mapping rules.

[0059] Step 7: Activate the corresponding accumulator based on the inspection results, accumulate the continuous inspection time and compare it with the preset threshold. If the threshold is reached, a completion signal is generated; otherwise, an audible and visual alarm is triggered.

[0060] In this embodiment of the invention, after correcting imaging distortion using an inverse projection transformation algorithm, the sub-pixel-level two-dimensional coordinates of key points are calculated using a convolutional neural network model, and the three-dimensional pose data is iteratively optimized using the PnP algorithm. Compared with traditional pose calculation methods based on pixel-level features, sub-pixel-level precision feature extraction effectively reduces the interference of image noise on the positioning results, reduces the pose calculation error of the product's axial reference point, and provides high-quality input for accurate calculation of spatial azimuth and pitch angles, thereby reducing the risk of misjudgment of the inspected surface from the source.

[0061] By employing a dual determination mechanism of a continuous inspection duration accumulator and a minimum detection duration threshold, invalid determinations caused by instantaneous image interference or operational jitter are avoided. Compared to existing technologies that rely solely on image sharpness for determination, this strategy comprehensively considers the spatiotemporal stability of the inspected surface.

[0062] In a preferred embodiment of the present invention, step 2, based on the coordinates of the reference point, drives a non-common-view multi-camera array to synchronously capture the image of the surface to be inspected. After distortion correction by inverse projection transformation, a convolutional neural network is used to calculate the sub-pixel-level two-dimensional coordinates of the key points on the imaging plane, including:

[0063] Step 21: Generate a spatial trigger signal based on the spatial coordinates of the reference point to drive the multi-camera array with non-common view layout to perform hardware-level synchronous exposure and generate a set of original images of the surface to be inspected with spatiotemporal alignment.

[0064] Step 22: Load the pre-calibrated camera intrinsic parameters and distortion parameters onto the original image set, map the pixel coordinates to the normalized plane through inverse projection transformation, and use a nonlinear compensation function to eliminate radial and tangential distortion to obtain a geometrically corrected image set.

[0065] Step 23: Define the feature search region based on the spatial coordinates of the reference point, perform multi-scale feature extraction on the geometrically corrected image set, and generate the pixel-level initial coordinates of the key points; using the pixel-level initial coordinates as the reference, calculate the gray-level gradient center offset vector in the local neighborhood, and calculate the sub-pixel-level two-dimensional coordinates of the key points.

[0066] Step 24: Establish a multi-view projection mapping relationship between the sub-pixel level two-dimensional coordinates and the reference point spatial coordinates, and calculate the reprojection error value; if the reprojection error value exceeds the tolerance threshold, adjust the feature search region parameters and re-execute; otherwise, generate the sub-pixel coordinate dataset.

[0067] In this embodiment of the invention, based on the spatial coordinates of the reference point X, Y, and Z in the product body coordinate system B, its position change in the global world coordinate system A is monitored in real time. When the reference point enters the preset field of view of any camera, a spatial trigger signal is generated. This signal is transmitted to the multi-camera array with a non-common view layout through a hardware synchronization bus (such as GPIO synchronization pulse), driving all cameras to perform exposure operations at the same time. The exposure time is dynamically adjusted according to the ambient light intensity (usually 5ms-50ms) to ensure that the synchronization error of the generated original image set of the surface to be inspected is controlled within 5μs in the time dimension, thus achieving spatiotemporal alignment.

[0068] The original image set is loaded with pre-calibrated camera intrinsic parameters (including focal length, principal point coordinates, etc.) and distortion parameters (radial distortion coefficient, tangential distortion coefficient). Through the inverse projection transformation algorithm, the two-dimensional coordinates of each pixel in the image are mapped to the normalized imaging plane of the camera optical center to eliminate the geometric distortion caused by perspective projection. For the residual distortion in the lens edge region, a nonlinear compensation function (distortion correction model based on polynomial fitting) is introduced to perform secondary correction on radial distortion (edge ​​pixel stretching / compression) and tangential distortion (image tilting and twisting), so that the residual distortion error of the geometrically corrected image set is controlled within 0.1 pixels.

[0069] The nonlinear compensation function is used to eliminate the radial and tangential distortions remaining after the inverse projection transformation. Its core is to establish a distortion compensation model through polynomial fitting. The specific process is as follows:

[0070] During the camera pre-calibration stage, a high-precision calibration board (such as a checkerboard or dot array) is used to cover the entire field of view of the camera and capture multiple sets of images to ensure that the sample points are evenly distributed in the central and edge areas of the image (the sample proportion in the edge area is not less than 40%). For each image, the theoretical projection coordinates (coordinates after inverse projection transformation) and the actual pixel coordinates of the corner points of the calibration board are extracted, and the difference between the two is calculated as the distortion error sample. The number of samples is adjusted according to the camera resolution (usually 500-2000 points).

[0071] A radial-tangential coupled polynomial model is used to correct residual distortion. The model input is the coordinates of the pixel in the normalized imaging plane (the preliminary corrected coordinates after inverse projection transformation), and the output is the distortion compensation offset. The polynomial order is set to 3-5 according to the lens distortion characteristics: 3rd order is used for low distortion lenses (such as industrial fixed-focus lenses), and 5th order is used for strong distortion lenses (such as wide-angle lenses). The feature variables include the radial distance from the pixel to the image center (reflecting radial distortion) and the polar angle (reflecting tangential distortion), and nonlinear mapping is achieved through the combination of polynomial terms.

[0072] The radial component includes square, fourth, and sixth power terms of the distance (corresponding to orders 3-5), while the tangential component includes the product of the distance and the polar angle sine / cosine, ensuring that the model simultaneously covers both "edge stretching" (radial) and "local tilting" (tangential) distortions. The model parameters are polynomial coefficient matrices with dimensions varying with the order: the 3rd order model contains 8-10 coefficients, and the 5th order model contains 15-20 coefficients, all determined through subsequent training.

[0073] Model training process: High-precision calibration board images (chessboard or dot array) captured by the camera are acquired. The calibration board plane is placed at multiple angles such as 0°, 30°, and 60° with the camera optical axis to cover different fields of view. The pixel coordinates of feature points on the calibration board are extracted through sub-pixel corner detection. The real-world coordinates are calculated by combining the physical size of the calibration board. The theoretical normalized coordinates (distortion-free ideal coordinates) are then obtained through camera intrinsic parameter transformation. The deviation between the actual pixel coordinates and the theoretical coordinates is used as training labels. The number of samples is set to 10,000-50,000 sets according to the lens distortion complexity, of which the sample proportion in the edge region is not less than 40% (because edge distortion is more significant).

[0074] Least squares iterative optimization:

[0075] Initial coefficient settings: The mean of the residual error of the inverse projection transformation is used as the initial compensation amount, and the initial value of the polynomial coefficients is set to 0 or a small range of random values ​​(-0.1 to 0.1) to avoid the iteration getting trapped in local optima.

[0076] Iterative process: In each iteration, the predicted compensation amount for all samples is calculated based on the current coefficients. The error equation is constructed using the least squares method (the goal is to minimize the sum of squares of the predicted bias and the true bias). The coefficients are updated using the gradient descent method. The learning rate is initially set to 0.01-0.1 and decreases by 50% every 20 iterations until the convergence condition is met.

[0077] Training stops when the sum of squared errors changes by less than 0.001 pixels over 10 consecutive iterations, or when the total number of iterations reaches 50-200. Finally, the coefficient combination with the smallest error in the validation set (accounting for 20% of the total samples) is retained.

[0078] Model Implementation and Compensation Calculation:

[0079] For each pixel in the geometrically corrected image set, its radial distance and polar angle to the image center are first calculated, and then substituted into the trained polynomial model to obtain the radial and tangential compensation offsets. The offsets are then superimposed on the coordinates after the inverse projection transformation to obtain the final corrected coordinates. To balance accuracy and efficiency, a regional calculation strategy is adopted: the central region of the image (with a radius within 1 / 3 of the field of view) is calculated quickly using a 3rd-order polynomial, while the edge regions are finely compensated using a 5th-order polynomial.

[0080] The absolute values ​​of the polynomial coefficients are controlled within the range of 0.001-0.1 (exceeding this range can easily lead to overcompensation); when calculating radial distance, the normalized planar coordinate range is limited to -1 to 1 (any part exceeding this range is considered an invalid pixel); the offset of the compensated pixel coordinates is controlled within 0.05-0.1 pixels to ensure the global consistency of the geometrically corrected image; radial distortion compensation: focuses on correcting the stretching / compression error of pixels along the radial direction, and the compensation amount changes nonlinearly with the increase of the distance from the pixel to the image center; corrects the offset error of pixels along the tangent direction, and the compensation amount is related to the intersection terms of the pixel coordinates (such as x·y).

[0081] Finally, a set of distortion compensation parameters (including polynomial coefficients and weighting factors) is generated for this camera.

[0082] Perform point-by-point compensation on the image pixels after inverse projection transformation: calculate the distance from each pixel to the image center and the coordinate intersection term; substitute it into the polynomial model to calculate the radial and tangential compensation amounts, and adjust the compensation amount range according to the degree of lens distortion (usually -1.5 pixels to +1.5 pixels); superimpose the compensation amount onto the coordinates after inverse projection transformation to obtain the corrected pixel coordinates, and complete the secondary correction of geometric distortion.

[0083] Randomly select 10%-20% of the calibration sample points for verification. If the average residual distortion error is ≤0.1 pixels, the compensation model is effective. If the error exceeds the limit, the polynomial order needs to be increased (not exceeding 4th order) or supplemented with edge region samples for refitting until the accuracy requirements are met.

[0084] Based on the spatial coordinates of the reference point X, Y, Z in the product's body coordinate system B, a feature search area is defined in the geometric correction image set: a rectangular search box is set with the theoretical projection position of the reference point as the center, and the size is adjusted according to the actual size of the product and the camera resolution (usually 32×32 pixels - 128×128 pixels).

[0085] Convolutional Neural Network Model Construction:

[0086] A lightweight feature extraction network is adopted, with an overall architecture consisting of 3-5 convolutional layers and 2-3 pooling layers. The input is a local image within the search area (size 224×224 pixels - 512×512 pixels), and the output is the pixel-level coordinate offset of key points. The first and second layers use 3×3 convolutional kernels (number 64-128) to extract low-level features such as edges and textures. The third to fifth layers use 5×5 convolutional kernels (number 128-256) to fuse multi-scale semantic features. Each convolutional layer is followed by a BatchNorm layer and a ReLU activation function to reduce overfitting and enhance non-linear expressive power.

[0087] The feature map is compressed to 2 channels (x, y coordinate offset) using a 1×1 convolution kernel, and pixel-level coordinate prediction values ​​are output using a linear activation function.

[0088] Model training process:

[0089] Product images under different lighting conditions (strong light, weak light, backlight) and different materials (metal, glass, plastic) were collected, and the subpixel coordinates of the reference points were manually labeled. The dataset was expanded to 100,000-500,000 samples through data augmentation (random rotation ±15°, scaling by 0.8-1.2 times, cropping of the region by 0.8-1.0 times, and adding Gaussian noise). The mean squared error (MSE) loss was used to calculate the Euclidean distance between the predicted coordinates and the labeled coordinates, focusing on optimizing the prediction accuracy of edge regions and reflective scenes. The Adam optimizer was used, with the initial learning rate set to 1e-4-1e-3, decaying to 1 / 10 of the original rate every 20 epochs. The batch size was 16-64, and the number of training epochs was 50-100. Training was stopped when the accuracy on the validation set showed no improvement for 5 consecutive epochs.

[0090] Model Implementation and Coordinate Calculation:

[0091] The trained model is quantized into FP16 format and deployed to edge computing devices (such as the NVIDIA Jetson series), with inference time controlled within 10ms.

[0092] The CNN model is used to extract features from the search area image and output the initial pixel-level coordinates of the key points (error ≤ 1 pixel).

[0093] Centered on the initial pixel-level coordinates, the gray-level gradient distribution is calculated in a local neighborhood of 5×5 pixels to 11×11 pixels. By analyzing the centroid offset law of gradient direction and magnitude, the sub-pixel-level offset vector is determined, improving the coordinate accuracy to the 0.01 pixel level, and finally obtaining the sub-pixel-level two-dimensional coordinates.

[0094] The subpixel-level 2D coordinates are associated with the spatial coordinates of the reference point in the product body coordinate system B to establish a projection mapping relationship under multiple views. The deviation (reprojection error) between each subpixel coordinate and the actual spatial coordinates of the reference point after backprojection to 3D space is calculated. A preset reprojection error tolerance threshold is set (0.5 pixels to 2 pixels according to the detection accuracy requirements): if the error exceeds the threshold, the size (expanded by 10%-30%) or position (offset by 5-10 pixels) of the feature search area is automatically adjusted, and the feature extraction and coordinate calculation in step 23 are re-executed. If the error is within the threshold range, a valid subpixel coordinate dataset is output.

[0095] Hardware-level synchronized exposure control effectively eliminates time differences between multiple cameras, avoiding feature misalignment caused by asynchronous shooting and laying the foundation for cross-view data fusion. A dual correction strategy combining inverse projection transformation and nonlinear compensation reduces the impact of lens optical distortion on feature localization, ensuring image geometric accuracy meets sub-pixel level computation requirements. Multi-scale feature extraction combined with grayscale gradient centroid analysis improves keypoint localization accuracy from pixel-level to sub-pixel-level, providing high-quality 2D coordinate input for subsequent 3D pose computation. A reprojection error verification mechanism effectively filters out abnormal coordinates caused by noise or mismatches by dynamically adjusting search parameters, improving the stability and reliability of the sub-pixel coordinate dataset.

[0096] In a preferred embodiment of the present invention, step 3, based on the sub-pixel two-dimensional coordinates and the spatial coordinates of coordinate system B, iteratively calculates the three-dimensional pose data of point O and points X, Y, and Z in each camera coordinate system using the PnP algorithm, including:

[0097] Step 31: Establish a projection correspondence set between two-dimensional imaging coordinates and three-dimensional spatial coordinates based on the sub-pixel coordinate dataset and the calibrated coordinate system B spatial coordinates; and parse the weighted representation relationship of spatial reference points based on the three-dimensional spatial point coordinates through a virtual control point mechanism.

[0098] Step 32: Based on the coordinates of the two-dimensional imaging points and the weighted representation relationship, construct the camera projection constraint equation; solve the projection constraint equation by matrix decomposition to obtain the coordinates of the virtual control points in the camera coordinate system; reconstruct the initial position of the spatial reference point in the camera coordinate system according to the coordinates of the virtual control points and the weighted representation relationship.

[0099] Step 33: Based on the initial position of the spatial reference point, calculate the pose parameters from the product coordinate system to the camera coordinate system; perform iterative optimization of the reprojection error value based on the initial position and pose parameters until convergence, and obtain the optimized 3D pose data of the camera coordinate system.

[0100] In this embodiment of the invention, a one-to-one 2D-3D point pair relationship set is established based on the subpixel coordinate dataset (two-dimensional coordinates of the imaging plane) and the three-dimensional spatial coordinates of the O, X, Y, and Z reference points calibrated in the product body coordinate system B. Each point pair contains subpixel-level two-dimensional coordinates (u, v) and corresponding three-dimensional coordinates (X_B, Y_B, Z_B), ensuring that each image contains at least 3 non-collinear reference points to meet the solution conditions of the PnP algorithm.

[0101] Virtual control point mechanism design:

[0102] Model Construction: A weighted fusion strategy is used to construct virtual control points. Spatial reference points O, X, Y, and Z are regarded as original control points. Two to three virtual control points are generated by analyzing their spatial topological relationships. The coordinates of the virtual control points are obtained by weighted summation of the coordinates of the original reference points. The weights are allocated according to the uniformity of the spatial distribution of the reference points (the weight values ​​range from 0.2 to 0.5, and the sum of all weights is 1). For example, if O is the origin and X / Y / Z are the axis endpoints, the virtual control points can be set as the midpoints of X and Y, Y and Z, etc., to enhance spatial geometric constraints.

[0103] By conducting preliminary experiments to statistically analyze the stability of different reference points under noise interference, higher weights (0.4-0.5) are assigned to reference points with high positioning accuracy (such as the Z point with clear edges), while lower weights (0.2-0.3) are assigned to reference points that are easily obstructed (such as the O point).

[0104] Solving projection constraint equations and reconstructing initial position:

[0105] Based on the pinhole camera model, a projection constraint equation is constructed by combining the weighted representation relationship between the coordinates (u, v) of the two-dimensional imaging point and the virtual control point. The equation uses the camera intrinsic parameters (focal length, principal point coordinates) as known quantities and the three-dimensional coordinates of the virtual control point in the camera coordinate system as unknowns, reflecting the projection mapping relationship from the three-dimensional spatial point to the two-dimensional imaging point.

[0106] Model Implementation: The projection constraint equations are solved using the Singular Value Decomposition (SVD) method. After transforming the equations into matrix form, singular values ​​are extracted by decomposing the coefficient matrix. The eigenvectors corresponding to the top three largest singular values ​​are retained to reconstruct the three-dimensional coordinates of the virtual control points in the camera coordinate system. During the decomposition process, a singular value threshold (ranging from 0.1 to 1.0) is set to filter out small singular values ​​caused by noise and avoid ill-conditioned solutions.

[0107] Perform a geometrical rationality check on the coordinates of the virtual control points obtained by the solution to ensure that the deviation between any two points and the theoretical value is within 5%; otherwise, readjust the matrix decomposition parameters.

[0108] Reconstruction of the initial position of the spatial reference point:

[0109] Based on the weighted representation relationship between the virtual control point coordinates and the original reference points (the weights defined in step 31), the initial three-dimensional coordinates of points O, X, Y, and Z in the camera coordinate system are calculated in reverse. For example, if the virtual control point P = w1X + w2Y, then X = (P - w2Y) / w1 (w1 ≠ 0). The initial position reconstruction of all reference points is completed in this way.

[0110] Based on the initial position of the spatial reference point, the pose parameters from the product coordinate system B to the camera coordinate system are calculated, including the rotation matrix (describing the attitude) and the translation vector (describing the position). The rotation matrix is ​​represented by a unit quaternion to avoid orthogonality constraints, and the translation vector is directly determined by the initial coordinates of the origin O.

[0111] Based on the current pose parameters, the three-dimensional coordinates of the reference point in coordinate system B are projected onto the imaging plane, and the Euclidean distance between the projected coordinates and the sub-pixel measured coordinates is calculated, i.e., the reprojection error.

[0112] Optimization process: The Levenberg-Marquardt (LM) iterative algorithm is used to minimize the reprojection error; the initial iteration step size is set to 0.01-0.1, and the step size is adjusted according to the error change after each iteration (the step size is reduced to 1 / 10 when the error increases, and increased to twice when the error decreases); the iteration termination condition is: the error change is less than 0.01 pixels for 3 consecutive iterations, or the number of iterations reaches the upper limit (the value range is 20-50 times).

[0113] After optimization, calculate the average reprojection error of all reference points. It must be controlled within 0.5 pixels. Otherwise, return to step 32 to solve for the virtual control point coordinates again.

[0114] After iterative convergence, the optimized rotation matrix and translation vector are output. Combined with the initial position of the reference point, the final 3D pose data of points O, X, Y, and Z in the camera coordinate system are generated.

[0115] The virtual control point mechanism reduces the impact of noise or occlusion of individual reference points on the solution results by weighted fusion of original reference point information, and enhances spatial geometric constraints, especially in the case of non-common-view camera layouts; the matrix factorization method combined with the singular value filtering strategy reduces the error between the virtual control point coordinates and the initial position of the reference points.

[0116] In a preferred embodiment of the present invention, step 4, based on the three-dimensional pose data, applies a pre-calibrated rigid transformation matrix to map to coordinate system A, and uses a spatial consistency fusion strategy to generate three-dimensional spatially consistent coordinates of point O and points X, Y, and Z in coordinate system A, including:

[0117] Step 41: Based on the 3D pose data, convert it into initial coordinates in global coordinate system A through a pre-calibrated rigid transformation matrix; based on the initial coordinates in coordinate system A and the time difference between camera exposure, calculate the motion offset and correct the coordinate position to generate a time-synchronized coordinate set;

[0118] Step 42: Calculate the fusion weight of each camera's observation data based on the reprojection error value of the pose calculation and the fluctuation range of the time-synchronized coordinate set; and generate the reference point fusion coordinates by performing a weighted average calculation based on the time-synchronized coordinate set and the fusion weight.

[0119] Step 43: Calculate the spacing between adjacent reference points in the fused coordinates, compare it with the original spacing of the calibrated coordinate system B, eliminate the spacing deviation through position offset iterative adjustment, and output the coordinates that meet the spacing consistency as the three-dimensional space consistent coordinates.

[0120] In this embodiment of the invention, based on the three-dimensional pose data in each camera coordinate system, a pre-calibrated rigid transformation matrix (containing rotation and translation parameters, obtained through offline calibration, with calibration error controlled within 0.1 mm) is loaded to batch convert the pose data into initial coordinates in the global world coordinate system A.

[0121] For the exposure time difference between non-co-view cameras (usually controlled within 5μs), the motion offset is estimated by combining the product motion characteristics: by analyzing the coordinate change rate of adjacent frames, the position offset corresponding to the time difference is calculated (the offset is usually less than 0.5mm in dynamic scenes), the initial coordinates are corrected point by point, and finally a time-synchronized coordinate set aligned with the time dimension is generated to ensure that the observation data of different cameras are consistent on the time base.

[0122] Fusion weight determination: Based on the reprojection error value output in step 3 (the reprojection error of a single camera is usually controlled within 0.5-2 pixels) and the fluctuation range of the time synchronization coordinate set (the amount of coordinate change in three consecutive frames), the reliability weight of each camera data is calculated. Camera data with smaller reprojection errors and lower fluctuation ranges are assigned higher weights, with weight values ​​ranging from 0.1 to 0.9, and the sum of the weights of all cameras is 1. For example, cameras with reprojection errors less than 1 pixel and fluctuation ranges less than 0.3 mm can have their weights set to 0.6-0.9; cameras with excessive errors or fluctuations have their weights reduced to 0.1-0.3.

[0123] Weighted averages are performed on points O, X, Y, and Z in the time-synchronized coordinate set: using the coordinate data of each camera as input, weighted summation is performed according to the corresponding weights to generate the fused coordinates of each reference point, eliminating coordinate deviations caused by the observation errors of a single camera.

[0124] Calculate the spatial distance between adjacent reference points in the fused coordinates (such as the distance between point O and point X, point O and point Y, and point O and point Z) and compare it with the original distance calibrated in the product body coordinate system B (the physical dimension calibrated offline, with an error of less than 0.05 mm). If the distance deviation exceeds the threshold (usually set to 1%-3% of the original distance), start the position offset iterative adjustment: with the original distance as the target, perform a small translation correction on the fused coordinates (each adjustment amount is less than 0.1 mm), repeat the calculation and correction of the distance, and the number of iterations is usually 3-5 times. When the distance deviation of all adjacent reference points is less than the threshold, output the final three-dimensional spatial consistency coordinates to ensure that the coordinate data conforms to the actual physical size constraints of the product.

[0125] By using a rigid transformation matrix and time-synchronized correction, the pose data of non-common-view cameras are unified into a global coordinate system, resolving the issue of inconsistent spatiotemporal references among multiple cameras and providing a reliable foundation for cross-view data fusion. A weighted fusion strategy based on errors and fluctuations dynamically allocates observation weights, reducing the impact of low-quality data on the fusion results and ensuring coordinate data stability even in the presence of camera noise or slight movement. A spacing consistency adjustment mechanism verifies coordinate rationality through physical size constraints, eliminating spatial distortion caused by accumulated errors and ensuring that the output coordinates match the actual product structure, providing a high-precision spatial reference for subsequent azimuth angle calculations. The entire process supports dynamic weight adjustment and iterative optimization, adapting to product movement in handheld scenarios and improving coordinate fusion accuracy under complex conditions.

[0126] In a preferred embodiment of the present invention, step 5, calculating the gaze vector from point O to the center of the pupil based on the pupil coordinates and spatial consistency coordinates; and calculating the product body axial vector from point O to each reference point based on the X, Y, and Z coordinates, includes:

[0127] Step 51: Obtain the three-dimensional spatial coordinates of the operator's pupil center in the global world coordinate system A, and obtain the three-dimensional spatial consistency coordinates of the product body coordinate system origin O and the reference points X, Y, Z in the global world coordinate system A.

[0128] Step 52: In the global world coordinate system A, perform spatial vector calculation based on the coordinates of the origin O and the pupil center to generate a three-dimensional gaze vector pointing from point O to the pupil center.

[0129] Step 53: In the global world coordinate system A, perform spatial vector calculations based on the coordinates of the origin O and the coordinates of the reference points X, Y, and Z, respectively, and generate the X-axis vector pointing from point O to reference point X, the Y-axis vector pointing to reference point Y, and the Z-axis vector pointing to reference point Z.

[0130] In this embodiment of the invention, the three-dimensional spatial coordinates of the operator's pupil center in the global world coordinate system A are obtained by an eye-tracking device (such as an infrared binocular camera). The coordinates are pre-calibrated and the error is controlled within the range of 0.5mm-2mm. The coordinates are updated every 50ms to adapt to slight head movements. The coordinate data of the product body coordinate system origin O and the reference points X, Y, and Z in coordinate system A are extracted from the three-dimensional spatial consistent coordinate set. The coordinates of point O are used as the reference origin for spatial vector calculation. The coordinates of points X, Y, and Z need to meet the orthogonality constraint with the calibration of coordinate system B (the perpendicular deviation between any two points in space is ≤0.5°).

[0131] In the global world coordinate system A, spatial vector calculation is performed with the coordinates of point O as the starting point and the coordinates of the pupil center as the ending point: the direction parameters of the vector are determined by coordinate difference analysis, covering the spatial pointing relationship in three dimensions: front-back, left-right, and up-down; after the calculation is completed, the vector is normalized to eliminate the interference of distance factors, so that the vector magnitude is uniformly 1 (unit vector), ensuring that the vector direction is comparable under different operating distances; the update frequency of the gaze vector is synchronized with the pupil coordinates (50ms / time), dynamically responding to changes in the viewing angle caused by the operator's head rotation or body displacement.

[0132] In the global world coordinate system A, starting from the coordinates of point O, spatial vector calculations are performed sequentially with the coordinates of the reference point X, Y, and Z:

[0133] X-axis vector: Based on the coordinate difference between point O and point X, a spatial vector is generated along the length of the product. Its magnitude is consistent with the physical length of the OX axis calibrated in coordinate system B (usually 5cm-20cm depending on the product type, such as the length of the OX axis of a mobile phone, which is about 6cm-15cm), and its direction is consistent with the length reference of the product body.

[0134] Y-axis vector: Based on the coordinate difference between point O and point Y, a spatial vector is generated along the product width direction. The length corresponds to the product width dimension (usually 3cm-15cm, such as the OY axis length of a mobile phone is about 4cm-10cm), and it is orthogonal to the X-axis vector (spatial angle deviation ≤1°).

[0135] Z-axis vector: Based on the coordinate difference between point O and point Z, a spatial vector is generated along the product thickness direction. The length corresponds to the product thickness (usually 0.5cm-5cm, such as the OZ axis length of a mobile phone, which is about 0.7cm-1.5cm), and it is orthogonal to both the X and Y axis vectors (spatial angle deviation ≤1°).

[0136] All axial vectors are normalized to ensure that directional features are independent of the actual product dimensions.

[0137] By frequently updating pupil coordinates and normalized gaze vectors, the system tracks the operator's gaze direction in real time, resolving spatial correlation failures caused by dynamic changes in viewing angles in handheld scenarios, and providing a precise viewing reference for surface inspection. Axial vectors calculated based on orthogonal reference points strictly maintain spatial orthogonality and are consistent with the physical dimensions of the product, ensuring that the vector direction accurately reflects the spatial orientation characteristics of the product's six sides and avoiding axial deviations caused by coordinate drift. The real-time updating and error control mechanism for coordinate data can adapt to scenarios involving slight operator tremors and body movements, reducing the impact of momentary interference on vector direction and ensuring the stability of subsequent orientation determination.

[0138] In a preferred embodiment of the present invention, step 6, based on the line-of-sight vector and the axis vector, calculates the azimuth and elevation angle features, segments the quadrants using a spatial cutting algorithm, and obtains the inspected surface result by combining a preset spatial orientation mapping rule, including:

[0139] Step 61: Calculate the projection vector on the horizontal plane of the global world coordinate system A based on the line of sight vector. Combine the projection directions of the X-axis vector and Y-axis vector of the product body coordinate system B on the horizontal plane to dynamically generate the reference azimuth reference axis. Calculate the azimuth feature between the projection vector and the reference azimuth reference axis.

[0140] Step 62: Calculate the angle between the line-of-sight vector and the horizontal projection vector to generate the pitch angle feature; perform azimuth quadrant segmentation based on the azimuth feature and pitch interval segmentation based on the pitch angle feature to form a combined spatial quadrant region.

[0141] In step 63, the combined spatial quadrant region is input into a preset spatial orientation mapping rule. The mapping rule associates the spatial quadrant with the correspondence between the product's hexahedral surface to obtain the test surface result.

[0142] The XY plane of the global world coordinate system A is defined as the horizontal plane. The line-of-sight vector (a three-dimensional vector pointing from point O to the center of the pupil) is projected onto the horizontal plane, and the Z-axis component is ignored to obtain a two-dimensional projection vector. At the same time, the projection directions of the X-axis vector (point O to the X reference point) and the Y-axis vector (point O to the Y reference point) of the product body coordinate system B onto the horizontal plane are extracted. The reference azimuth reference axis is dynamically generated according to the spatial proportion of the two: if the projection magnitude of the X-axis is greater than that of the Y-axis, the X-axis projection is used as the main reference axis; otherwise, the Y-axis projection is used as the main reference axis; if the two are close, the composite vector is taken as the reference axis. The angle between the projection vector and the reference azimuth reference axis is calculated as the azimuth feature quantity, with a value range of 0°-360°, increasing in a clockwise direction.

[0143] The angle between the line-of-sight vector and the horizontal projection vector is calculated as the pitch angle feature, with a value range of -90° to 90° (positive for upward and negative for downward). Based on the azimuth feature, azimuth quadrant segmentation is performed: 0°-360° is divided into 4 equal quadrants, each covering 90° (e.g., 0°-90° is the first quadrant, 90°-180° is the second quadrant, etc.). Based on the pitch angle feature, pitch interval segmentation is performed: with 0° as the boundary, it is divided into an upper interval (0°-90°) and a lower interval (-90°-0°). Through the combination of azimuth quadrants and pitch intervals, 8 combined spatial quadrant regions are formed (4 azimuths × 2 pitches), each region corresponding to a specific spatial azimuth range.

[0144] A pre-defined spatial orientation mapping rule is established to create a correspondence between the combined spatial quadrant regions and the six-sided surfaces of the product. The rule is based on the axial definition of the product's body coordinate system B. For example, when the line of sight vector falls into the first orientation quadrant and the pitch angle is in the upper interval, it corresponds to the "front" of the product; when it falls into the second orientation quadrant and the pitch angle is in the lower interval, it corresponds to the "right side" of the product, etc. The mapping rule needs to cover all six surfaces. For the boundary conditions of adjacent regions (such as an azimuth angle of 90° and a pitch angle of 0°), the "nearest associativity" principle is used to determine the dominant region. The combined spatial quadrant regions obtained in step 62 are input into the mapping rule, and the unique corresponding inspected surface results are output (such as "front", "back", "left side", "right side", "top", and "bottom").

[0145] By dynamically generating a reference azimuth axis, the azimuth angle calculation adapts to changes in product posture, resolving the positioning deviation issue when the fixed reference axis is used and the product is tilted. The azimuth angle judgment error is controlled within a preset quadrant range. A combined segmentation strategy of azimuth quadrants and pitch intervals is employed, covering a 360° omnidirectional range and a ±90° pitch range. This accurately distinguishes the different orientations of the six sides in space, adapting to any rotational posture of the manually held product. Preset mapping rules are based on the axial characteristics of the product's coordinate system, avoiding subjective judgment bias and ensuring consistent judgment standards for the inspected surface under different operators or testing scenarios, thus reducing the misjudgment rate. The "nearest assignment" principle is used to handle feature quantity boundary cases, reducing judgment fluctuations caused by angle thresholds and improving the stability of the testing results.

[0146] In a preferred embodiment of the present invention, step 7, activating the corresponding accumulator based on the inspected surface result, accumulating the continuous inspection time and comparing it with a preset threshold, generates a completion signal if the threshold is reached, otherwise triggers an audible and visual alarm, including:

[0147] Step 71: Based on the results of the inspected surface, retrieve the current value of the continuous inspection duration accumulator corresponding to the inspected surface; add the current value of the accumulator to the time step value of the current detection cycle to generate the updated accumulated duration;

[0148] Determine whether the updated cumulative duration reaches the preset minimum detection duration threshold;

[0149] Step 72: If the threshold is reached, a completion signal for the inspected surface is generated and the accumulator is locked; if the threshold is not reached, the accumulator update value is stored as the input reference for the next detection cycle; when a change in the inspected surface result is detected, the original accumulator update value for the inspected surface is retrieved; if the update value is less than the minimum detection time threshold, an audible and visual alarm device is triggered and the accumulator is reset; if the threshold has been reached, the completion status is maintained and no alarm is triggered.

[0150] In this embodiment of the invention, based on the inspected surface result (such as "front", "side X", etc.) output in step 6, the current value of the continuous inspection duration accumulator corresponding to the inspected surface is retrieved from the system storage module (the accumulator value is 0 in the initial state); the detection system executes the detection process according to a fixed cycle, and the time step value of the current detection cycle is set according to the real-time requirements (usually 20ms-200ms, to ensure that the response speed and calculation efficiency are balanced); the current value of the accumulator is added to the time step value to generate the updated accumulated duration (for example: the current value is 800ms, the time step is 200ms, and the updated value is 1000ms).

[0151] A minimum detection time threshold is preset (set to 1s-3s according to product detection requirements to ensure that the inspected surface is stably displayed). The updated cumulative time is compared with this threshold to determine whether the completion conditions are met.

[0152] Completion signal generation: If the updated cumulative duration reaches or exceeds the minimum detection duration threshold, the system automatically generates a completion signal for the inspected surface (e.g., marked as "detected"), and locks the accumulator (stops subsequent value updates) to avoid duplicate counting.

[0153] If the accumulated time does not reach the threshold, the updated accumulated time is stored in the system cache as the initial value of the accumulator for the next detection cycle, and the time accumulation continues; when the system detects a change in the result of the inspected surface (such as switching from "front" to "side Y"), it immediately retrieves the updated value of the original inspected surface accumulator:

[0154] If the original accumulator value is less than the minimum detection time threshold, it is determined that the valid detection has not been completed, and the audible and visual alarm device is triggered (such as the buzzer sounds and the warning light flashes). At the same time, the original accumulator value of the inspected surface is reset to 0. If the original accumulator value has reached the threshold, it is maintained in the "completed" state, no alarm is triggered, and only the initial value of the accumulator of the new inspected surface is reset to 0 and a new timer is started.

[0155] By employing a continuous duration accumulation and threshold judgment mechanism, false completion judgments caused by momentary display or operational jitter are avoided, ensuring that each inspected surface is stably inspected for a sufficient duration, thus improving the reliability of the results. An audible and visual alarm device promptly alerts the operator when an inspected surface has not completed effective inspection, reducing the risk of missed inspections; the completion status locking mechanism prevents duplicate inspections, improving process efficiency. The configurability of the time step and threshold (20ms-200ms step, 1s-3s threshold) adapts to the inspection needs of different products, accommodates differences in operator habits, and enhances system flexibility. The accumulator's independent timing and reset logic for each surface ensures accurate status recording during multi-surface inspection, avoids timing interference between different inspected surfaces, and simplifies the complexity of system status management.

Claims

1. A method for determining the six-sided completion status of a handheld product based on cross-view image fusion, characterized in that, The method includes: Step 1: Establish a global world coordinate system A and calibrate the three-dimensional coordinates of the operator's pupil center; establish a product body coordinate system B and calibrate the three-dimensional coordinates of the origin O and the orthogonal axial reference points X, Y, and Z. Step 2: Based on the reference point coordinates, drive the non-common-view multi-camera array to synchronously capture the image of the surface to be inspected. After the distortion is corrected by inverse projection transformation, the sub-pixel level two-dimensional coordinates of the key points on the imaging plane are calculated by a convolutional neural network. Step 3: Based on the sub-pixel 2D coordinates and the spatial coordinates of coordinate system B, the PnP algorithm is used to iteratively calculate the 3D pose data of point O and points X, Y, and Z in each camera coordinate system. Step 4: Based on the 3D pose data, apply a pre-calibrated rigid transformation matrix to map to coordinate system A, and use a spatial consistency fusion strategy to generate the 3D spatially consistent coordinates of point O and points X, Y, and Z in coordinate system A. Step 5: Based on the pupil coordinates and spatial consistency coordinates, calculate the line-of-sight vector from point O to the center of the pupil; based on the X, Y, and Z coordinates, calculate the product body axial vector from point O to each reference point. Step 6: Based on the line-of-sight vector and axis vector, calculate the azimuth and pitch angle features, divide the quadrants using a spatial cutting algorithm, and obtain the test surface result by combining the preset spatial orientation mapping rules. Step 7: Activate the corresponding accumulator based on the inspection results, accumulate the continuous inspection time and compare it with the preset threshold. If the threshold is reached, a completion signal is generated; otherwise, an audible and visual alarm is triggered.

2. The method for determining the six-sided completion status of a handheld product based on cross-view image fusion according to claim 1, characterized in that, Step 2 includes: Step 21: Generate a spatial trigger signal based on the spatial coordinates of the reference point to drive the multi-camera array with non-common view layout to perform hardware-level synchronous exposure and generate a set of original images of the surface to be inspected with spatiotemporal alignment. Step 22: Load the pre-calibrated camera intrinsic parameters and distortion parameters onto the original image set, map the pixel coordinates to the normalized plane through inverse projection transformation, and use a nonlinear compensation function to eliminate radial and tangential distortion to obtain a geometrically corrected image set. Step 23: Define the feature search region based on the spatial coordinates of the reference point, perform multi-scale feature extraction on the geometrically corrected image set, and generate the pixel-level initial coordinates of the key points; using the pixel-level initial coordinates as the reference, calculate the gray-level gradient center offset vector in the local neighborhood, and calculate the sub-pixel-level two-dimensional coordinates of the key points. Step 24: Establish a multi-view projection mapping relationship between the sub-pixel level two-dimensional coordinates and the reference point spatial coordinates, and calculate the reprojection error value; if the reprojection error value exceeds the tolerance threshold, adjust the feature search region parameters and re-execute; otherwise, generate the sub-pixel coordinate dataset.

3. The method for determining the six-sided completion status of a handheld product based on cross-view image fusion according to claim 2, characterized in that, Based on sub-pixel 2D coordinates and spatial coordinates of coordinate system B, the PnP algorithm iteratively calculates the 3D pose data of point O and points X, Y, and Z in each camera coordinate system, including: Step 31: Establish a projection correspondence set between two-dimensional imaging coordinates and three-dimensional spatial coordinates based on the sub-pixel coordinate dataset and the calibrated coordinate system B spatial coordinates; and parse the weighted representation relationship of spatial reference points based on the three-dimensional spatial point coordinates through a virtual control point mechanism. Step 32: Based on the coordinates of the two-dimensional imaging points and the weighted representation relationship, construct the camera projection constraint equation; solve the projection constraint equation by matrix decomposition to obtain the coordinates of the virtual control points in the camera coordinate system; reconstruct the initial position of the spatial reference point in the camera coordinate system according to the coordinates of the virtual control points and the weighted representation relationship. Step 33: Based on the initial position of the spatial reference point, calculate the pose parameters from the product coordinate system to the camera coordinate system; perform iterative optimization of the reprojection error value based on the initial position and pose parameters until convergence, and obtain the optimized 3D pose data of the camera coordinate system.

4. The method for determining the six-sided completion status of a handheld product based on cross-view image fusion according to claim 3, characterized in that, Based on 3D pose data, a pre-calibrated rigid transformation matrix is ​​applied and mapped to coordinate system A. A spatial consistency fusion strategy is used to generate 3D spatially consistent coordinates of point O and points X, Y, and Z in coordinate system A, including: Step 41: Based on the 3D pose data, convert it into initial coordinates in global coordinate system A through a pre-calibrated rigid transformation matrix; based on the initial coordinates in coordinate system A and the time difference between camera exposure, calculate the motion offset and correct the coordinate position to generate a time-synchronized coordinate set; Step 42: Calculate the fusion weight of each camera's observation data based on the reprojection error value of the pose calculation and the fluctuation range of the time-synchronized coordinate set; and generate the reference point fusion coordinates by performing a weighted average calculation based on the time-synchronized coordinate set and the fusion weight. Step 43: Calculate the spacing between adjacent reference points in the fused coordinates, compare it with the original spacing of the calibrated coordinate system B, eliminate the spacing deviation through position offset iterative adjustment, and output the coordinates that meet the spacing consistency as the three-dimensional space consistent coordinates.

5. The method for determining the six-sided completion status of a handheld product based on cross-view image fusion according to claim 4, characterized in that, Based on the pupil coordinates and the spatial consistency coordinates from step 4, calculate the gaze vector from point O to the pupil center; based on the X, Y, and Z coordinates, calculate the product body axial vectors from point O to each reference point, including: Step 51: Obtain the three-dimensional spatial coordinates of the operator's pupil center in the global world coordinate system A, and obtain the three-dimensional spatial consistency coordinates of the product body coordinate system origin O and the reference points X, Y, Z in the global world coordinate system A. Step 52: In the global world coordinate system A, perform spatial vector calculation based on the coordinates of the origin O and the pupil center to generate a three-dimensional gaze vector pointing from point O to the pupil center. Step 53: In the global world coordinate system A, perform spatial vector calculations based on the coordinates of the origin O and the coordinates of the reference points X, Y, and Z, respectively, and generate the X-axis vector pointing from point O to reference point X, the Y-axis vector pointing to reference point Y, and the Z-axis vector pointing to reference point Z.

6. The method for determining the six-sided completion status of a handheld product based on cross-view image fusion according to claim 5, characterized in that, Based on the line-of-sight vector and axis vector, azimuth and elevation angle features are calculated. Quadrants are segmented using a spatial cutting algorithm, and the inspected surface is obtained by combining this with preset spatial orientation mapping rules. This includes: Step 61: Calculate the projection vector on the horizontal plane of the global world coordinate system A based on the line of sight vector. Combine the projection directions of the X-axis vector and Y-axis vector of the product body coordinate system B on the horizontal plane to dynamically generate the reference azimuth reference axis. Calculate the azimuth feature between the projection vector and the reference azimuth reference axis. Step 62: Calculate the angle between the line-of-sight vector and the horizontal projection vector to generate the pitch angle feature; perform azimuth quadrant segmentation based on the azimuth feature and pitch interval segmentation based on the pitch angle feature to form a combined spatial quadrant region. In step 63, the combined spatial quadrant region is input into a preset spatial orientation mapping rule. The mapping rule associates the spatial quadrant with the correspondence between the product's hexahedral surface to obtain the test surface result.

7. The method for determining the six-sided completion status of a handheld product based on cross-view image fusion according to claim 1, characterized in that, Based on the results of the inspected surface, the corresponding accumulator is activated, the continuous inspection time is accumulated and compared with a preset threshold. If the threshold is reached, a completion signal is generated; otherwise, an audible and visual alarm is triggered, including: Step 71: Based on the results of the inspected surface, retrieve the current value of the continuous inspection duration accumulator corresponding to the inspected surface; add the current value of the accumulator to the time step value of the current detection cycle to generate the updated accumulated duration; Determine whether the updated cumulative duration reaches the preset minimum detection duration threshold; Step 72: If the threshold is reached, a completion signal for the inspected surface is generated and the accumulator is locked; if the threshold is not reached, the accumulator update value is stored as the input reference for the next detection cycle; when a change in the inspected surface result is detected, the original accumulator update value for the inspected surface is retrieved; if the update value is less than the minimum detection time threshold, an audible and visual alarm device is triggered and the accumulator is reset; if the threshold has been reached, the completion status is maintained and no alarm is triggered.

8. The method for determining the six-sided completion status of a handheld product based on cross-view image fusion according to claim 7, characterized in that, The original image set is loaded with pre-calibrated camera intrinsic parameters and distortion parameters. Pixel coordinates are mapped to a normalized plane using inverse projection transformation. Radial and tangential distortions are eliminated using a nonlinear compensation function, resulting in a geometrically corrected image set, including: Step 81: Obtain the two-dimensional coordinates of each pixel in the original image set and load the pre-calibrated camera intrinsic parameter matrix; construct the inverse projection mapping relationship from the pixel coordinate system to the normalized imaging plane based on the camera intrinsic parameter matrix; apply the inverse projection mapping relationship to perform coordinate transformation calculation on each pixel to generate the corresponding normalized plane coordinates; aggregate the normalized plane coordinates of all pixels to generate an uncorrected normalized coordinate set. Step 82: Based on the uncorrected normalized coordinate set, load the pre-calibrated radial distortion parameters and tangential distortion parameters, apply the nonlinear compensation function to calculate the distortion offset, and generate the corrected normalized coordinate set. Step 83: Based on the corrected normalized coordinate set, reconstruct the geometric position in the pixel coordinate system through the inverse mapping relationship to generate a geometrically corrected image set.

9. The method for determining the six-sided completion status of a handheld product based on cross-view image fusion according to claim 8, characterized in that, Based on the uncorrected normalized coordinate set, pre-calibrated radial and tangential distortion parameters are loaded, and a nonlinear compensation function is applied to calculate the distortion offset, generating a corrected normalized coordinate set, including: Step 91: Load the pre-calibrated radial distortion parameter sequence and tangential distortion parameter sequence, and obtain the uncorrected normalized coordinate set; for each coordinate point in the normalized coordinate set, calculate the squared radial distance from it to the origin of the normalized plane; based on the radial distortion parameter sequence and the squared radial distance, calculate the radial distortion scaling factor. Step 92: Multiply the radial distortion scaling factor with the original coordinate point coordinate value to generate the radial distortion offset component; calculate the coordinate product term and the radius square term based on the tangential distortion parameter sequence and the original coordinate point coordinate value; sum the coordinate product term and the radius square term according to the tangential distortion parameter to generate the tangential distortion offset component. Step 93: The radial distortion offset component and the tangential distortion offset component are vector-superimposed to generate the total distortion offset; the total distortion offset is subtracted from the original uncorrected coordinates to generate the corrected normalized coordinates; all corrected coordinate points are aggregated to generate the corrected normalized coordinate set.

10. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 9.

Images