Method for judging six-surface completion state of handheld product based on cross view angle image fusion

Abstract

The invention provides a handheld product six-surface completion state judgment method based on cross view angle image fusion, and relates to the technical field of data processing, and the method comprises the steps: calculating a sight line vector of a point O pointing to a pupil center based on pupil coordinates and spatial consistency coordinates; calculating a product body axial vector of the point O pointing to each datum point based on the coordinates of the points X, Y and Z; on the basis of the sight line vector and the axial vector, azimuth angle and pitch angle characteristic quantities are calculated, quadrant segmentation is carried out through a space cutting algorithm, and a detected surface result is obtained in combination with a preset space azimuth mapping rule; activating a corresponding accumulator based on a detected surface result, accumulating continuous detected duration and comparing the accumulated duration with a preset threshold value; if the duration reaches the threshold value, generating a completion signal, otherwise, triggering an audible and visual alarm. According to the method, the detection reliability and the result credibility in a complex operation scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for determining the completion status of six surfaces of a handheld product based on cross-view image fusion. Background Art

[0002] In the manufacturing of consumer electronics and precision instruments, six-sided appearance quality and assembly integrity testing of handheld products (such as smartphones, smartwatches, and portable sensors) is a critical step in ensuring product reliability. Taking smartphones as an example, testing six features, such as the fit of the front and rear covers, the tightness of the frame screws, and the integrity of the interfaces, directly impacts user experience and brand reputation. Traditional inspection methods sometimes rely on manual inspection of individual products under a fixed light source. This single-machine inspection is time-consuming and subject to operator fatigue and experience differences, resulting in a high rate of missed defects.

[0003] Some existing models use independent calibration methods to obtain the intrinsic parameters of each camera, without establishing a unified coordinate system. This leads to systematic deviations in the pose data of different cameras. For example, there are significant errors in the coordinate conversion between cameras. When the product moves, the 3D pose solution error accumulates with distance, ultimately leading to incorrect recognition of the inspected surface. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for determining the completion status of six sides of a handheld product based on cross-view image fusion, thereby improving the detection reliability and result credibility in complex operation scenarios.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows: In a first aspect, a method for determining the completion status of six sides of a handheld product based on cross-view image fusion is provided, the method comprising: Step 1: Establish a global world coordinate system A and calibrate the three-dimensional coordinates of the operator's pupil center; establish a product body coordinate system B and calibrate the three-dimensional coordinates of the origin O and the orthogonal axial reference points X, Y, and Z; Step 2: Based on the coordinates of the reference points, a non-co-view multi-camera array is driven to synchronously capture the image of the surface to be inspected. After distortion correction through inverse projection transformation, a convolutional neural network is used to calculate the sub-pixel two-dimensional coordinates of the key points on the imaging plane. Step 3: Based on the sub-pixel 2D coordinates and the spatial coordinates of coordinate system B, the PnP algorithm is used to iteratively calculate the 3D pose data of point O and points X, Y, and Z in each camera coordinate system. Step 4: Based on the 3D pose data, the pre-calibrated rigid transformation matrix is ​​applied to map to the coordinate system A, and the spatial consistency fusion strategy is used to generate the 3D spatial consistency coordinates of point O and points X, Y, and Z in the coordinate system A; Step 5: Calculate the sight line vector from point O to the pupil center based on the pupil coordinates and the spatial consistency coordinates; calculate the product body axial vector from point O to each reference point based on the X, Y, and Z coordinates; Step 6: Based on the sight vector and the axial vector, calculate the azimuth and elevation angle feature quantities, divide the quadrants using the spatial cutting algorithm, and obtain the inspected surface result in combination with the preset spatial orientation mapping rules; Step 7: Based on the inspection results, the corresponding accumulator is activated to accumulate the continuous inspection time and compare it with the preset threshold. If the threshold is reached, a completion signal is generated; otherwise, an audible and visual alarm is triggered.

[0006] Furthermore, the step 2 includes: Step 21: Generate a spatial trigger signal based on the spatial coordinates of the reference point to drive the multi-camera array in a non-common view layout to perform hardware-level synchronous exposure, thereby generating a set of original images of the surface to be inspected that are aligned in time and space. Step 22: Load the pre-calibrated camera intrinsic parameters and distortion parameters into the original image set, map the pixel coordinates to the normalized plane through inverse projection transformation, and use a nonlinear compensation function to eliminate radial and tangential distortion to obtain a geometrically corrected image set. Step 23: Define a feature search area based on the spatial coordinates of the reference points, perform multi-scale feature extraction on the geometrically corrected image set, and generate pixel-level initial coordinates of the key points; using the pixel-level initial coordinates as a reference, calculate the grayscale gradient centroid offset vector in the local neighborhood to calculate the sub-pixel two-dimensional coordinates of the key points; In step 24, a multi-view projection mapping relationship is established between the sub-pixel two-dimensional coordinates and the reference point spatial coordinates, and the reprojection error value is calculated; if the reprojection error value exceeds the tolerance threshold, the feature search area parameters are adjusted and the search is re-executed; otherwise, a sub-pixel coordinate dataset is generated.

[0007] Furthermore, based on the sub-pixel two-dimensional coordinates and the spatial coordinates of coordinate system B, the three-dimensional pose data of point O and points X, Y, and Z in each camera coordinate system are iteratively calculated using the PnP algorithm, including: Step 31: establishing a projection correspondence relationship set between two-dimensional imaging coordinates and three-dimensional spatial coordinates based on the sub-pixel coordinate data set and the calibrated coordinate system B space coordinates; and resolving the weighted representation relationship of the spatial reference points through the virtual control point mechanism according to the three-dimensional spatial point coordinates. Step 32: construct a camera projection constraint equation based on the relationship between the coordinates of the two-dimensional imaging points and the weighted representation; solve the projection constraint equation using a matrix decomposition method to obtain the coordinates of the virtual control points in the camera coordinate system; and reconstruct the initial position of the spatial reference point in the camera coordinate system based on the relationship between the coordinates of the virtual control points and the weighted representation; Step 33: Based on the initial position of the spatial reference point, calculate the pose parameters from the product coordinate system to the camera coordinate system; perform iterative optimization of the reprojection error value based on the initial position and pose parameters until convergence, and obtain the optimized three-dimensional pose data of the camera coordinate system.

[0008] Furthermore, based on the 3D pose data, the pre-calibrated rigid transformation matrix is ​​applied to map to the coordinate system A, and a spatial consistency fusion strategy is used to generate the 3D spatial consistency coordinates of point O and points X, Y, and Z in the coordinate system A, including: Step 41: Based on the three-dimensional pose data, the initial coordinates in the global coordinate system A are converted using a pre-calibrated rigid transformation matrix; based on the initial coordinates in the coordinate system A and the camera exposure time difference, the motion offset is calculated and the coordinate position is corrected to generate a time-synchronized coordinate set; Step 42: Calculate the fusion weight of each camera observation data based on the reprojection error value of the pose solution and the fluctuation amplitude of the time-synchronized coordinate set; perform weighted average calculation based on the fusion weight based on the time-synchronized coordinate set to generate the reference point fusion coordinates; Step 43: Calculate the distance between adjacent reference points in the fused coordinates, compare it with the original distance in the calibrated coordinate system B, eliminate the distance deviation through iterative adjustment of position offset, and output the coordinates that meet the distance consistency as the three-dimensional space consistent coordinates.

[0009] Furthermore, based on the pupil coordinates and the spatial consistency coordinates, the sight line vector from point O to the pupil center is calculated; based on the X, Y, and Z coordinates, the axial vector of the product body from point O to each reference point is calculated, including: Step 51: Obtain the calibrated three-dimensional coordinates of the operator's pupil center in the global world coordinate system A, and obtain the three-dimensional consistent coordinates of the origin O of the product body coordinate system and the reference points X, Y, and Z in the global world coordinate system A; Step 52: In the global world coordinate system A, perform space vector calculation based on the coordinates of the origin O and the pupil center to generate a three-dimensional sight line vector from point O to the pupil center; Step 53, in the global world coordinate system A, perform space vector calculation based on the coordinates of the origin O and the X, Y, and Z coordinates of the reference points, and generate an X-axis vector pointing from point O to the reference point X, a Y-axis vector pointing to the reference point Y, and a Z-axis vector pointing to the reference point Z.

[0010] Furthermore, based on the sight vector and the axial vector, the azimuth and elevation angle features are calculated, the quadrants are divided by the spatial cutting algorithm, and the results of the examined surface are obtained by combining the preset spatial orientation mapping rules, including: Step 61: Calculate the projection vector on the horizontal plane of the global world coordinate system A based on the sight line vector, dynamically generate a reference orientation reference axis based on the projection direction of the X-axis vector and the Y-axis vector of the product body coordinate system B on the horizontal plane, and calculate the azimuth angle feature between the projection vector and the reference orientation reference axis; Step 62: Calculate the angle between the sight line vector and the horizontal projection vector to generate a pitch angle feature; perform azimuth quadrant segmentation based on the azimuth feature, and perform pitch interval segmentation based on the pitch angle feature to form a combined spatial quadrant area; In step 63, the combined spatial quadrant area is input into a preset spatial orientation mapping rule, and the mapping rule associates the corresponding relationship between the spatial quadrant and the surface of the product hexahedron to obtain the inspection surface result.

[0011] Furthermore, based on the inspection results, the corresponding accumulator is activated to accumulate the continuous inspection time and compare it with the preset threshold. If the threshold is reached, a completion signal is generated. Otherwise, an audible and visual alarm is triggered, including: Step 71: Based on the inspection result of the inspection surface, retrieve the current value of the continuous inspection time accumulator corresponding to the inspection surface; increase the current value of the accumulator by the time step value of the current inspection cycle to generate an updated accumulated time; Determine whether the updated cumulative duration reaches the preset minimum detection duration threshold; Step 72: If the threshold is reached, a completion signal of the inspected surface is generated and the accumulator is locked; if the threshold is not reached, the updated value of the accumulator is stored as the input reference for the next detection cycle; when a change in the inspected surface result is detected, the updated value of the original inspected surface accumulator is retrieved; if the updated value is less than the minimum detection time threshold, the sound and light alarm device is triggered and the accumulator is reset; if the threshold is reached, the completion status is maintained and no alarm is triggered.

[0012] Furthermore, the pre-calibrated camera intrinsic parameters and distortion parameters are loaded into the original image set, the pixel coordinates are mapped to the normalized plane through inverse projection transformation, and the radial and tangential distortions are eliminated using a nonlinear compensation function to obtain a geometrically corrected image set, including: Step 81: Obtain the two-dimensional coordinates of the pixel coordinate system of each pixel point in the original image set and load the pre-calibrated camera intrinsic parameter matrix; construct an inverse projection mapping relationship from the pixel coordinate system to the normalized imaging plane based on the camera intrinsic parameter matrix; apply the inverse projection mapping relationship to perform coordinate transformation calculation on each pixel point to generate the corresponding normalized plane coordinates; aggregate the normalized plane coordinates of all pixel points to generate an uncorrected normalized coordinate set; Step 82 , based on the uncorrected normalized coordinate set, load the pre-calibrated radial distortion parameters and tangential distortion parameters, apply a nonlinear compensation function to calculate the distortion offset, and generate a corrected normalized coordinate set; Step 83 : Based on the corrected normalized coordinate set, reconstruct the geometric position in the pixel coordinate system through an inverse mapping relationship to generate a geometrically corrected image set.

[0013] Furthermore, based on the uncorrected normalized coordinate set, the pre-calibrated radial distortion parameters and tangential distortion parameters are loaded, and the distortion offset is calculated by applying a nonlinear compensation function to generate a corrected normalized coordinate set, including: Step 91: Load the pre-calibrated radial distortion parameter sequence and tangential distortion parameter sequence, and obtain an uncorrected normalized coordinate set; for each coordinate point in the normalized coordinate set, calculate the square of the radial distance from the origin of the normalized plane; and calculate the radial distortion scaling factor based on the radial distortion parameter sequence and the square of the radial distance. Step 92: Multiply the radial distortion scaling factor by the coordinate value of the original coordinate point to generate a radial distortion offset component; calculate the coordinate product term and the radius square term based on the tangential distortion parameter sequence and the coordinate value of the original coordinate point; and sum the coordinate product term and the radius square term weighted by the tangential distortion parameter to generate a tangential distortion offset component. Step 93 , vector superposition of the radial distortion offset component and the tangential distortion offset component is performed to generate a total distortion offset; the total distortion offset is subtracted from the original uncorrected coordinates to generate corrected normalized coordinates; and all corrected coordinate points are aggregated to generate a corrected normalized coordinate set.

[0014] The above solution of the present invention includes at least the following beneficial effects: This method uses a back-projection transformation algorithm to correct imaging distortion, then uses a convolutional neural network model to calculate the sub-pixel 2D coordinates of key points. This is then combined with a PnP algorithm to iteratively optimize the 3D pose data. Compared to traditional pose calculation methods based on pixel-level features, this sub-pixel feature extraction effectively reduces the interference of image noise on positioning results, minimizing the pose calculation error of the product's axial reference points. This provides high-quality input for the precise calculation of spatial azimuth and pitch angles, essentially reducing the risk of misjudgment of the inspected surface.

[0015] This dual-determination mechanism, combining a continuous inspection duration accumulator and a minimum inspection duration threshold, avoids invalid determinations caused by transient image interference or operational jitter. Compared to existing technologies that rely solely on image clarity, this strategy comprehensively considers the spatiotemporal stability of the inspected surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a method for determining the completion status of six sides of a handheld product based on cross-view image fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0018] like Figure 1 As shown, an embodiment of the present invention proposes a method for determining the completion status of six sides of a handheld product based on cross-view image fusion, the method comprising the following steps: Step 1: Establish a global world coordinate system A and calibrate the three-dimensional coordinates of the operator's pupil center; establish a product body coordinate system B and calibrate the three-dimensional coordinates of the origin O and the orthogonal axial reference points X, Y, and Z; Step 2: Based on the coordinates of the reference points, a non-co-view multi-camera array is driven to synchronously capture the image of the surface to be inspected. After distortion correction through inverse projection transformation, a convolutional neural network is used to calculate the sub-pixel two-dimensional coordinates of the key points on the imaging plane. Step 3: Based on the sub-pixel 2D coordinates and the spatial coordinates of coordinate system B, the PnP algorithm is used to iteratively calculate the 3D pose data of point O and points X, Y, and Z in each camera coordinate system. Step 4: Based on the 3D pose data, the pre-calibrated rigid transformation matrix is ​​applied to map to the coordinate system A, and the spatial consistency fusion strategy is used to generate the 3D spatial consistency coordinates of point O and points X, Y, and Z in the coordinate system A; Step 5: Calculate the sight line vector from point O to the pupil center based on the pupil coordinates and the spatial consistency coordinates; calculate the product body axial vector from point O to each reference point based on the X, Y, and Z coordinates; Step 6: Based on the sight vector and the axial vector, calculate the azimuth and elevation angle feature quantities, divide the quadrants using the spatial cutting algorithm, and obtain the inspected surface result in combination with the preset spatial orientation mapping rules; Step 7: Based on the inspection results, the corresponding accumulator is activated to accumulate the continuous inspection time and compare it with the preset threshold. If the threshold is reached, a completion signal is generated; otherwise, an audible and visual alarm is triggered.

[0019] In an embodiment of the present invention, after correcting imaging distortion using an inverse projection transformation algorithm, a convolutional neural network model is used to calculate the sub-pixel 2D coordinates of key points. This is then combined with a PnP algorithm to iteratively optimize the 3D pose data. Compared to traditional pose calculation methods based on pixel-level features, sub-pixel feature extraction effectively reduces the interference of image noise on positioning results, minimizing the pose calculation error of the product's axial reference points. This provides high-quality input for the precise calculation of spatial azimuth and pitch angles, essentially reducing the risk of misjudgment of the inspected surface.

[0020] This dual-determination mechanism, combining a continuous inspection duration accumulator and a minimum inspection duration threshold, avoids invalid determinations caused by transient image interference or operational jitter. Compared to existing technologies that rely solely on image clarity, this strategy comprehensively considers the spatiotemporal stability of the inspected surface.

[0021] In a preferred embodiment of the present invention, step 2, based on the coordinates of the reference points, drives a non-co-view multi-camera array to synchronously capture an image of the surface to be inspected, corrects the distortion through inverse projection transformation, and uses a convolutional neural network to calculate the sub-pixel two-dimensional coordinates of the key points on the imaging plane, including: Step 21: Generate a spatial trigger signal based on the spatial coordinates of the reference point to drive the multi-camera array in a non-common view layout to perform hardware-level synchronous exposure, thereby generating a set of original images of the surface to be inspected that are aligned in time and space. Step 22: Load the pre-calibrated camera intrinsic parameters and distortion parameters into the original image set, map the pixel coordinates to the normalized plane through inverse projection transformation, and use a nonlinear compensation function to eliminate radial and tangential distortion to obtain a geometrically corrected image set. Step 23: Define a feature search area based on the spatial coordinates of the reference points, perform multi-scale feature extraction on the geometrically corrected image set, and generate pixel-level initial coordinates of the key points; using the pixel-level initial coordinates as a reference, calculate the grayscale gradient centroid offset vector in the local neighborhood to calculate the sub-pixel two-dimensional coordinates of the key points; In step 24, a multi-view projection mapping relationship is established between the sub-pixel two-dimensional coordinates and the reference point spatial coordinates, and the reprojection error value is calculated; if the reprojection error value exceeds the tolerance threshold, the feature search area parameters are adjusted and the search is re-executed; otherwise, a sub-pixel coordinate dataset is generated.

[0022] In an embodiment of the present invention, based on the spatial coordinates of the reference point X, Y, and Z in the product body coordinate system B, its position change in the global world coordinate system A is monitored in real time. When the reference point enters the preset field of view of any camera, a spatial trigger signal is generated; this signal is transmitted to the multi-camera array in a non-common view layout via a hardware synchronization bus (such as a GPIO synchronization pulse), driving all cameras to perform exposure operations at the same time. The exposure time is dynamically adjusted according to the ambient light intensity (usually 5ms-50ms), ensuring that the synchronization error of the generated original image set of the surface to be inspected in the time dimension is controlled within 5μs, thereby achieving spatiotemporal alignment.

[0023] The original image set is loaded with the pre-calibrated camera intrinsic parameters (including focal length, principal point coordinates, etc.) and distortion parameters (radial distortion coefficient, tangential distortion coefficient). Through the inverse projection transformation algorithm, the two-dimensional coordinates of each pixel in the image are mapped to the normalized imaging plane of the camera optical center to eliminate the geometric deformation caused by perspective projection. To address the residual distortion in the edge area of ​​the lens, a nonlinear compensation function (a distortion correction model based on polynomial fitting) is introduced to perform secondary corrections for radial distortion (edge ​​pixel stretching / compression) and tangential distortion (image tilt and distortion), ultimately controlling the distortion residual error of the geometrically corrected image set to within 0.1 pixel.

[0024] The nonlinear compensation function is used to eliminate the radial and tangential distortions remaining after the inverse projection transformation. Its core is to establish a distortion compensation model through polynomial fitting. The specific process is as follows: During the camera pre-calibration stage, use a high-precision calibration plate (such as a chessboard or dot array) to cover the entire field of view of the camera and capture multiple sets of images to ensure that the sample points are evenly distributed in the center and edge areas of the image (the proportion of samples in the edge area is not less than 40%). For each image, the theoretical projection coordinates (coordinates after inverse projection transformation) and actual pixel coordinates of the corner points of the calibration plate are extracted, and the difference between the two is calculated as the distortion error sample. The number of samples is adjusted according to the camera resolution (usually 500-2000 points). A radial-tangential coupled polynomial model is used to correct residual distortion. The model input is the coordinates of the pixel point in the normalized imaging plane (the preliminary corrected coordinates after inverse projection transformation), and the output is the distortion compensation offset. The polynomial order is set to 3-5 according to the lens distortion characteristics: 3rd order is used for low-distortion lenses (such as industrial fixed-focus lenses), and 5th order is used for high-distortion lenses (such as wide-angle lenses). The characteristic variables include the radial distance from the pixel point to the image center (reflecting radial distortion) and the polar angle (reflecting tangential distortion). Nonlinear mapping is achieved through the combination of polynomial terms. The radial component contains the square, fourth power, and sixth power of the distance (corresponding to orders 3-5), and the tangential component contains the product of the distance and the sine / cosine of the polar angle, ensuring that the model covers both "edge stretching" (radial) and "local tilt" (tangential) distortions. The model parameters are polynomial coefficient matrices, and the dimensions vary with the order: the third-order model contains 8-10 coefficients, and the fifth-order model contains 15-20 coefficients, all of which are determined through subsequent training.

[0025] Model training process: High-precision calibration plate images (chessboard or dot array) taken by the camera are collected. The calibration plate plane is placed at multiple angles such as 0°, 30°, and 60° to the camera optical axis to cover different field of view areas. The pixel coordinates of the calibration plate feature points are extracted through sub-pixel corner detection, and their real-world coordinates are calculated based on the physical dimensions of the calibration plate. The theoretical normalized coordinates (ideal coordinates without distortion) are then converted through the camera's intrinsic parameters. The deviation between the actual pixel coordinates and the theoretical coordinates is used as the training label. The number of samples is set to 10,000-50,000 groups based on the complexity of lens distortion, of which the proportion of edge area samples is not less than 40% (because edge distortion is more significant). Least squares iterative optimization: Initial coefficient setting: The mean of the residual error of the inverse projection transformation is used as the initial compensation amount, and the initial value of the polynomial coefficient is set to 0 or a random value in a small range (-0.1 to 0.1) to avoid the iteration falling into the local optimum. Iterative process: In each iteration, the predicted compensation amount of all samples is calculated based on the current coefficient, and the error equation is constructed using the least squares method (the goal is to minimize the sum of the squares of the predicted deviation and the true deviation); the coefficients are updated using the gradient descent method, and the learning rate is initially set to 0.01-0.1, and decays by 50% every 20 iterations until the convergence conditions are met. Training is stopped when the change in the sum of squared errors for 10 consecutive iterations is less than 0.001 pixels, or when the total number of iterations reaches 50-200; the coefficient combination with the smallest error in the validation set (accounting for 20% of the total samples) is finally retained. Model implementation and compensation calculation: For each pixel in the geometrically corrected image set, the radial distance and polar angle to the image center are first calculated, and then substituted into the trained polynomial model to obtain the radial and tangential compensation offsets; the offsets are superimposed on the coordinates after the inverse projection transformation to obtain the final corrected coordinates; to balance accuracy and efficiency, a regional calculation strategy is adopted: the image center area (radius within 1 / 3 of the field of view) is quickly calculated using a third-order polynomial, and the edge area is finely compensated using a fifth-order polynomial. The absolute values ​​of the polynomial coefficients are controlled within the range of 0.001-0.1 (exceeding this range can easily lead to overcompensation). When calculating radial distance, the normalized plane coordinate range is limited to -1 to 1 (the portion exceeding this range is considered invalid pixels). The deviation of the compensated pixel coordinates is controlled within 0.05-0.1 pixels to ensure the global consistency of the geometrically corrected image. Radial distortion compensation focuses on correcting the radial stretching / compression errors of pixels, and the compensation amount varies nonlinearly with the increase in the distance from the pixel to the image center. The offset error of pixels along the tangential direction is corrected, and the compensation amount is related to the cross term of the pixel coordinates (such as x·y). Finally, a distortion compensation parameter set (including polynomial coefficients and weight factors) for the camera is generated.

[0026] Point-by-point compensation is performed on the image pixels after the inverse projection transformation: for each pixel, the distance from the image center and the coordinate cross term are calculated; the radial and tangential compensation amounts are substituted into the polynomial model, and the compensation value range is adjusted according to the degree of lens distortion (usually -1.5 pixels to +1.5 pixels); the compensation amount is superimposed on the coordinates after the inverse projection transformation to obtain the corrected pixel coordinates, completing the secondary correction of geometric distortion. 10%-20% of the calibration sample points are randomly selected for verification. If the average residual distortion error is ≤0.1 pixel, the compensation model is valid. If the error exceeds the limit, the polynomial order needs to be increased (not exceeding 4th order) or samples in the edge area need to be added and refitted until the accuracy requirements are met. Based on the spatial coordinates of the reference point X, Y, and Z in the product coordinate system B, define a feature search area in the geometrically corrected image set: set a rectangular search box centered on the theoretical projection position of the reference point, and adjust the size according to the actual size of the product and the camera resolution (usually 32×32 pixels to 128×128 pixels). Convolutional neural network model construction: A lightweight feature extraction network is used, and the overall architecture consists of 3-5 convolutional layers and 2-3 pooling layers. The input is a local image within the search area (size 224×224 pixels to 512×512 pixels), and the output is the pixel-level coordinate offset of the key points. The 1st to 2nd layers use 3×3 convolution kernels (number 64-128) to extract underlying features such as edges and textures. The 3rd to 5th layers use 5×5 convolution kernels (number 128-256) to fuse multi-scale semantic features. After each convolution layer, a BatchNorm layer and ReLU activation function are connected to reduce overfitting and enhance nonlinear expression capabilities. The feature map is compressed into 2 channels (x, y coordinate offsets) through a 1×1 convolution kernel, and a linear activation function is used to output pixel-level coordinate prediction values. Model training process: Product images were collected under different lighting conditions (strong light, weak light, backlight) and materials (metal, glass, plastic), and the sub-pixel coordinates of the reference points were manually annotated as labels. The dataset was expanded to 100,000-500,000 samples through data augmentation (random rotation of ±15°, scaling by 0.8-1.2 times, cropping by 0.8-1.0 times, and adding Gaussian noise). The mean squared error (MSE) loss was used to calculate the Euclidean distance between the predicted and annotated coordinates, with a focus on optimizing the prediction accuracy of edge regions and reflective scenes. The Adam optimizer was used with an initial learning rate set of 1e-4-1e-3, which was decayed to 1 / 10 of the original rate every 20 rounds. The batch size was 16-64, and the number of training rounds was 50-100. Training was terminated after no improvement in the validation set accuracy for five consecutive rounds. Model implementation and coordinate solution: The trained model is quantized into FP16 format and deployed to edge computing devices (such as the NVIDIA Jetson series), with inference time controlled within 10ms. The CNN model is used to extract features from the search area image and output the initial pixel-level coordinates of the key points (error ≤ 1 pixel). Taking the initial pixel coordinates as the center, the grayscale gradient distribution is calculated in a local neighborhood of 5×5 pixels to 11×11 pixels. By analyzing the centroid offset law of the gradient direction and amplitude, the sub-pixel offset vector is determined, and the coordinate accuracy is improved to 0.01 pixel level, ultimately obtaining sub-pixel two-dimensional coordinates. Associate the sub-pixel two-dimensional coordinates with the spatial coordinates of the reference point in the product coordinate system B to establish a projection mapping relationship under multiple perspectives; calculate the deviation (reprojection error) of each sub-pixel coordinate from the actual spatial coordinates of the reference point after back-projection into three-dimensional space; preset the reprojection error tolerance threshold (set to 0.5 pixels-2 pixels based on the detection accuracy requirements): If the error exceeds the threshold, automatically adjust the size of the feature search area (expand by 10%-30%) or position (offset by 5-10 pixels), and re-execute the feature extraction and coordinate solution in step 23; if the error is within the threshold, output a valid sub-pixel coordinate data set.

[0027] Hardware-level synchronized exposure control effectively eliminates time differences between multiple cameras, avoiding feature misalignment caused by asynchronous shooting and laying the foundation for cross-view data fusion. A dual correction strategy combining inverse projection transformation with nonlinear compensation reduces the impact of lens optical distortion on feature positioning, ensuring that image geometric accuracy meets sub-pixel resolution requirements. Multi-scale feature extraction combined with grayscale gradient centroid analysis improves key point positioning accuracy from the pixel level to the sub-pixel level, providing high-quality two-dimensional coordinate input for subsequent three-dimensional pose resolution. The reprojection error correction mechanism dynamically adjusts search parameters to effectively filter out anomalous coordinates caused by noise or mismatching, improving the stability and reliability of sub-pixel coordinate datasets.

[0028] In a preferred embodiment of the present invention, step 3, iteratively calculating the three-dimensional pose data of point O and points X, Y, and Z in each camera coordinate system using a PnP algorithm based on the sub-pixel two-dimensional coordinates and the spatial coordinates of coordinate system B, includes: Step 31: establishing a projection correspondence relationship set between two-dimensional imaging coordinates and three-dimensional spatial coordinates based on the sub-pixel coordinate data set and the calibrated coordinate system B space coordinates; and resolving the weighted representation relationship of the spatial reference points through the virtual control point mechanism according to the three-dimensional spatial point coordinates. Step 32: construct a camera projection constraint equation based on the relationship between the coordinates of the two-dimensional imaging points and the weighted representation; solve the projection constraint equation using a matrix decomposition method to obtain the coordinates of the virtual control points in the camera coordinate system; and reconstruct the initial position of the spatial reference point in the camera coordinate system based on the relationship between the coordinates of the virtual control points and the weighted representation; Step 33: Based on the initial position of the spatial reference point, calculate the pose parameters from the product coordinate system to the camera coordinate system; perform iterative optimization of the reprojection error value based on the initial position and pose parameters until convergence, and obtain the optimized three-dimensional pose data of the camera coordinate system.

[0029] In an embodiment of the present invention, a one-to-one corresponding 2D-3D point pair relationship set is established based on the sub-pixel coordinate data set (two-dimensional coordinates of the imaging plane) and the three-dimensional spatial coordinates of the O, X, Y, and Z reference points calibrated in the product body coordinate system B; each point pair contains sub-pixel two-dimensional coordinates (u, v) and corresponding three-dimensional coordinates (X_B, Y_B, Z_B), ensuring that each image contains at least 3 non-collinear reference points to meet the solution conditions of the PnP algorithm. Virtual control point mechanism design: Model construction: A weighted fusion strategy is used to construct virtual control points. The spatial reference points O, X, Y, and Z are regarded as original control points. By analyzing their spatial topological relationship, 2-3 virtual control points are generated. The coordinates of the virtual control points are obtained by weighted summation of the coordinates of the original reference points. The weights are assigned according to the uniformity of the spatial distribution of the reference points (the weight range is 0.2-0.5, and the sum of all weights is 1). For example, if O is the origin and X / Y / Z are the axis endpoints, the virtual control points can be set to the midpoint of X and Y, the midpoint of Y and Z, etc., to enhance spatial geometric constraints. Through preliminary experiments, we statistically analyzed the stability of different reference points under noise interference, and assigned higher weights (0.4-0.5) to reference points with high positioning accuracy (such as point Z with clear edges), and lower weights (0.2-0.3) to reference points that are easily occluded (such as point O). Solving the projection constraint equations and reconstructing the initial position: Based on the pinhole camera model, the projection constraint equation is constructed by combining the weighted representation relationship between the two-dimensional imaging point coordinates (u, v) and the virtual control points. The equation uses the camera intrinsic parameters (focal length, principal point coordinates) as known quantities and the three-dimensional coordinates of the virtual control points in the camera coordinate system as unknown quantities, reflecting the projection mapping relationship from three-dimensional space points to two-dimensional imaging points. Model implementation: The projection constraint equation is solved using the matrix singular value decomposition (SVD) method. After converting the equation into matrix form, the singular values ​​are extracted by decomposing the coefficient matrix. The eigenvectors corresponding to the first three largest singular values ​​are retained to reconstruct the three-dimensional coordinates of the virtual control points in the camera coordinate system. During the decomposition process, a singular value threshold (ranging from 0.1 to 1.0) is set to filter out small singular values ​​caused by noise and avoid pathological solutions. The geometric rationality of the obtained virtual control point coordinates is checked to ensure that the deviation between the distance between any two points and the theoretical value is within 5%. Otherwise, the matrix decomposition parameters are readjusted. Reconstruction of the initial position of the spatial reference point: Based on the weighted representation relationship between the virtual control point coordinates and the original reference points (the weights defined in step 31), reversely calculate the initial three-dimensional coordinates of points O, X, Y, and Z in the camera coordinate system. For example, if the virtual control point P = w1X + w2Y, then X = (P - w2Y) / w1 (w1 ≠ 0), and so on to complete the reconstruction of the initial positions of all reference points. Based on the initial position of the spatial reference point, the pose parameters from the product coordinate system B to the camera coordinate system are calculated, including the rotation matrix (describing the posture) and the translation vector (describing the position); the rotation matrix is ​​represented by a unit quaternion to avoid orthogonality constraints, and the translation vector is directly determined by the initial coordinates of the origin O.

[0030] According to the current pose parameters, the three-dimensional coordinates of the reference point in coordinate system B are projected onto the imaging plane, and the Euclidean distance between the projected coordinates and the sub-pixel measured coordinates, i.e., the reprojection error, is calculated. Optimization process: The Levenberg-Marquardt (LM) iterative algorithm was used to minimize the reprojection error. The initial iteration step size was set to 0.01-0.1, and the step size was adjusted after each iteration based on the error change (reduced to 1 / 10 when the error increased and increased to 2 times when the error decreased). The iteration termination condition was: the error change for three consecutive iterations was less than 0.01 pixels, or the number of iterations reached the upper limit (range 20-50 times). After the optimization is completed, the average reprojection error of all reference points is calculated and must be controlled within 0.5 pixels. Otherwise, return to step 32 to re-solve the coordinates of the virtual control points. After the iteration converges, the optimized rotation matrix and translation vector are output and combined with the initial position of the reference point to generate the final three-dimensional pose data of the O, X, Y, and Z points in the camera coordinate system. The virtual control point mechanism reduces the impact of noise or occlusion of a single reference point on the solution results by weighted fusion of the original reference point information, especially enhancing the spatial geometric constraints under non-common view camera layout; the matrix decomposition method combined with the singular value filtering strategy reduces the error between the virtual control point coordinates and the initial position of the reference point.

[0031] In a preferred embodiment of the present invention, step 4, based on the three-dimensional pose data, applies a pre-calibrated rigid transformation matrix to map to the coordinate system A, and adopts a spatial consistency fusion strategy to generate the three-dimensional spatial consistency coordinates of the point O and the X, Y, and Z points in the coordinate system A, including: Step 41: Based on the three-dimensional pose data, the initial coordinates in the global coordinate system A are converted using a pre-calibrated rigid transformation matrix; based on the initial coordinates in the coordinate system A and the camera exposure time difference, the motion offset is calculated and the coordinate position is corrected to generate a time-synchronized coordinate set; Step 42: Calculate the fusion weight of each camera observation data based on the reprojection error value of the pose solution and the fluctuation amplitude of the time-synchronized coordinate set; perform weighted average calculation based on the fusion weight based on the time-synchronized coordinate set to generate the reference point fusion coordinates; Step 43: Calculate the distance between adjacent reference points in the fused coordinates, compare it with the original distance in the calibrated coordinate system B, eliminate the distance deviation through iterative adjustment of position offset, and output the coordinates that meet the distance consistency as the three-dimensional space consistent coordinates.

[0032] In an embodiment of the present invention, based on the three-dimensional pose data in each camera coordinate system, a pre-calibrated rigid transformation matrix (including rotation parameters and translation parameters, obtained through offline calibration, and the calibration error is controlled within 0.1 mm) is loaded to batch convert the pose data into the initial coordinates in the global world coordinate system A. For the exposure time difference between non-common view cameras (usually controlled within 5μs), the motion offset is estimated based on the product's motion characteristics: by analyzing the rate of change of coordinates of adjacent frames, the position offset corresponding to the time difference is calculated (the offset is usually less than 0.5mm in dynamic scenes), and the initial coordinates are corrected point by point. Finally, a time-synchronized coordinate set aligned in the time dimension is generated to ensure that the observation data of different cameras are consistent on the time basis. Fusion weight determination: Calculate the credibility weight of each camera data based on the reprojection error value output in step 3 (the reprojection error of a single camera is usually controlled within 0.5-2 pixels) and the fluctuation amplitude of the time-synchronized coordinate set (the amount of coordinate change in three consecutive frames). Camera data with smaller reprojection errors and lower fluctuation amplitudes are assigned higher weights. The weight range is 0.1-0.9, and the sum of all camera weights is 1. For example, a camera with a reprojection error less than 1 pixel and a fluctuation amplitude less than 0.3mm can be set to a weight of 0.6-0.9. The weight of a camera with an error or fluctuation exceeding the standard is reduced to 0.1-0.3. A weighted average is performed on the O, X, Y, and Z points in the time-synchronized coordinate set. The coordinate data of each camera is used as input, and weighted summation is performed according to the corresponding weight to generate the fused coordinates of each reference point, eliminating the coordinate deviation caused by the observation error of a single camera. The spatial spacing between adjacent reference points in the fused coordinates (e.g., the distances between point O and point X, point O and point Y, and point O and point Z) is calculated and compared with the original spacing calibrated in the product's coordinate system B (offline calibrated physical dimensions with an error of less than 0.05mm). If the spacing deviation exceeds a threshold (typically set to 1%-3% of the original spacing), an iterative position offset adjustment is initiated: using the original spacing as the target, a micro-translation correction (each adjustment is less than 0.1mm) is made to the fused coordinates. The spacing is then repeatedly calculated and corrected, typically three to five times. When the spacing deviations of all adjacent reference points are less than the threshold, the final 3D spatial consistency coordinates are output to ensure that the coordinate data conforms to the actual physical size constraints of the product. Through rigid transformation matrices and time synchronization correction, the pose data of non-co-view cameras are unified into the global coordinate system, solving the problem of inconsistent spatiotemporal references among multiple cameras and providing a reliable foundation for cross-view data fusion. A weighted fusion strategy based on errors and fluctuations dynamically allocates observation weights, reducing the impact of low-quality data on the fusion results and ensuring that coordinate data remains stable in the presence of camera noise or slight movement. The spacing consistency adjustment mechanism verifies the rationality of coordinates through physical size constraints, eliminating spatial distortion caused by accumulated errors, ensuring that the output coordinates are consistent with the actual structure of the product, and providing a high-precision spatial reference for subsequent azimuth angle calculations. The entire process supports dynamic weight adjustment and iterative optimization, which can adapt to product movement in manual handheld scenarios and improve coordinate fusion accuracy under complex working conditions. In a preferred embodiment of the present invention, step 5, calculating the sight line vector from point O to the pupil center based on the pupil coordinates and the spatial consistency coordinates; and calculating the product body axial vector from point O to each reference point based on the X, Y, and Z coordinates, includes: Step 51: Obtain the calibrated three-dimensional coordinates of the operator's pupil center in the global world coordinate system A, and obtain the three-dimensional consistent coordinates of the origin O of the product body coordinate system and the reference points X, Y, and Z in the global world coordinate system A; Step 52: In the global world coordinate system A, perform space vector calculation based on the coordinates of the origin O and the pupil center to generate a three-dimensional sight line vector from point O to the pupil center; Step 53, in the global world coordinate system A, perform space vector calculation based on the coordinates of the origin O and the X, Y, and Z coordinates of the reference points, and generate an X-axis vector pointing from point O to the reference point X, a Y-axis vector pointing to the reference point Y, and a Z-axis vector pointing to the reference point Z.

[0033] In an embodiment of the present invention, the three-dimensional spatial coordinates of the operator's pupil center in the global world coordinate system A are obtained through an eye tracking device (such as an infrared binocular camera). After pre-calibration, the error of the coordinates is controlled within the range of 0.5mm-2mm and is updated every 50ms to adapt to slight movements of the head. The coordinate data of the origin O of the product body coordinate system and the reference points X, Y, and Z in the coordinate system A are extracted from the three-dimensional spatial consistency coordinate set, where the coordinates of point O serve as the reference origin for spatial vector calculation, and the coordinates of points X, Y, and Z must meet the orthogonality constraint calibrated with the coordinate system B (the spatial pairwise vertical deviation is ≤0.5°).

[0034] In the global world coordinate system A, spatial vector calculations are performed with the coordinates of point O as the starting point and the coordinates of the pupil center as the end point. The directional parameters of the vector are determined through coordinate difference analysis, covering the spatial pointing relationships in the three dimensions of front-back, left-right, and up-down. After the calculation is completed, the vector is normalized to eliminate interference from distance factors, so that the vector modulus is unified to 1 (unit vector), ensuring that the vector direction at different operating distances is comparable. The update frequency of the line of sight vector is synchronized with the pupil coordinates (50ms / time), dynamically responding to changes in perspective caused by the operator's head rotation or body displacement. In the global world coordinate system A, take the coordinates of point O as the starting point and perform space vector calculations with the coordinates of the reference points X, Y, and Z in sequence: X-axis vector: Generates a spatial vector along the length of the product based on the coordinate difference between points O and X. Its modulus is consistent with the physical length of the OX axis calibrated in coordinate system B (depending on the product type, usually 5cm-20cm, such as the OX axis length of a mobile phone is approximately 6cm-15cm), and its direction remains consistent with the length reference of the product itself. Y-axis vector: Based on the coordinate difference between point O and point Y, a spatial vector along the width of the product is generated. The module length corresponds to the product width (usually 3cm-15cm, such as the OY axis length of a mobile phone is about 4cm-10cm), and it maintains an orthogonal relationship with the X-axis vector (spatial angle deviation ≤ 1°). Z-axis vector: Based on the coordinate difference between point O and point Z, a spatial vector along the product thickness direction is generated. The mold length corresponds to the product thickness dimension (usually 0.5cm-5cm, such as the OZ axis length of a mobile phone is about 0.7cm-1.5cm), and it is orthogonal to the X and Y axis vectors (spatial angle deviation ≤ 1°). All axial vectors are normalized to ensure that directional characteristics are independent of the actual product size. Through high-frequency updates of pupil coordinates and normalized gaze vectors, the operator's gaze direction is tracked in real time, resolving the spatial correlation failure caused by dynamic changes in perspective in manual handheld scenarios and providing an accurate perspective reference for determining the inspected surface. The axial vectors calculated based on orthogonal reference points strictly maintain spatial orthogonality and are consistent with the physical dimensions of the product itself, ensuring that the vector direction truly reflects the spatial orientation characteristics of the six sides of the product and avoiding axial deviations caused by coordinate drift. The real-time update of coordinate data and the error control mechanism can adapt to scenarios such as slight operator shaking and body movement, reducing the impact of instantaneous interference on the vector direction and ensuring the stability of subsequent orientation determination.

[0035] In a preferred embodiment of the present invention, step 6, based on the sight vector and the axial vector, calculates the azimuth and elevation angle feature quantities, divides the quadrants using a spatial cutting algorithm, and obtains the examined surface result in combination with a preset spatial orientation mapping rule, including: Step 61: Calculate the projection vector on the horizontal plane of the global world coordinate system A based on the sight line vector, dynamically generate a reference orientation reference axis based on the projection direction of the X-axis vector and the Y-axis vector of the product body coordinate system B on the horizontal plane, and calculate the azimuth angle feature between the projection vector and the reference orientation reference axis; Step 62: Calculate the angle between the sight line vector and the horizontal projection vector to generate a pitch angle feature; perform azimuth quadrant segmentation based on the azimuth feature, and perform pitch interval segmentation based on the pitch angle feature to form a combined spatial quadrant area; In step 63, the combined spatial quadrant area is input into a preset spatial orientation mapping rule, and the mapping rule associates the corresponding relationship between the spatial quadrant and the surface of the product hexahedron to obtain the inspection surface result. Define the XY plane of the global world coordinate system A as the horizontal plane, project the sight vector (a three-dimensional vector pointing from point O to the center of the pupil) onto the horizontal plane, and ignore the Z-axis component to obtain a two-dimensional projection vector. At the same time, extract the projection directions of the X-axis vector (point O pointing to the X reference point) and the Y-axis vector (point O pointing to the Y reference point) of the product body coordinate system B on the horizontal plane. Dynamically generate a reference orientation reference axis based on the spatial proportions of the two: if the X-axis projection modulus is greater than the Y-axis, the X-axis projection is used as the primary reference axis; otherwise, the Y-axis projection is used as the primary reference axis. If the two are close, the composite vector is used as the reference axis. Calculate the angle between the projection vector and the reference orientation reference axis as the azimuth feature, with a value range of 0°-360°, increasing in a clockwise direction. The angle between the sight line vector and the horizontal plane projection vector is calculated as the pitch angle feature, with a value range of -90°-90° (positive when the sight line is pointing upward and negative when the sight line is pointing downward). Azimuth quadrant segmentation is performed based on the azimuth angle feature: 0°-360° is divided into 4 equal quadrants, each covering 90° (such as 0°-90° is the first quadrant, 90°-180° is the second quadrant, etc.). Pitch interval segmentation is performed based on the pitch angle feature: with 0° as the boundary, it is divided into the upper interval (0°-90°) and the lower interval (-90°-0°). By combining the azimuth quadrant and the pitch interval, 8 combined spatial quadrant areas are formed (4 azimuths × 2 pitches), each of which corresponds to a specific spatial orientation range. Preset spatial orientation mapping rules to establish the correspondence between the combined spatial quadrant area and the hexahedral surface of the product: the rules are based on the axial definition of the product body coordinate system B. For example, when the line of sight vector falls into the first azimuth quadrant and the pitch angle is in the upper interval, it corresponds to the "front" of the product; when it falls into the second azimuth quadrant and the pitch angle is in the lower interval, it corresponds to the "right side" of the product, etc.; the mapping rules need to cover all 6 surfaces, and for the boundary conditions of adjacent areas (such as azimuth angle 90° and pitch angle 0°), the "nearest belonging" principle is used to determine the dominant area; the combined spatial quadrant area obtained in step 62 is input into the mapping rules, and the unique corresponding inspected surface result (such as "front", "back", "left side", "right side", "top", and "bottom") is output.

[0036] By dynamically generating a reference azimuth reference axis, the azimuth angle calculation adapts to changes in the product's posture, solving the positioning deviation problem of the fixed reference axis when the product is tilted, and the azimuth angle judgment error is controlled within the preset quadrant range. A combined segmentation strategy of azimuth quadrant and pitch interval is adopted, covering 360° in all directions and ±90° pitch range, which can accurately distinguish the different orientations of the six sides in space and adapt to any rotation posture of the manually held product. The preset mapping rules are based on the axial characteristics of the product's body coordinate system to avoid subjective judgment deviations, unify the judgment standards of the inspected surfaces under different operators or different inspection scenarios, and reduce the misjudgment rate. The boundary conditions of the feature quantities are handled by the "nearest attribution" principle to reduce the judgment fluctuations caused by the angle critical value and improve the stability of the inspection results. In a preferred embodiment of the present invention, step 7, based on the inspection result, activates the corresponding accumulator, accumulates the continuous inspection time and compares it with a preset threshold. If the threshold is reached, a completion signal is generated. Otherwise, an audible and visual alarm is triggered, including: Step 71: Based on the inspection result of the inspection surface, retrieve the current value of the continuous inspection time accumulator corresponding to the inspection surface; increase the current value of the accumulator by the time step value of the current inspection cycle to generate an updated accumulated time; Determine whether the updated cumulative duration reaches the preset minimum detection duration threshold; Step 72: If the threshold is reached, a completion signal of the inspected surface is generated and the accumulator is locked; if the threshold is not reached, the updated value of the accumulator is stored as the input reference for the next detection cycle; when a change in the inspected surface result is detected, the updated value of the original inspected surface accumulator is retrieved; if the updated value is less than the minimum detection time threshold, the sound and light alarm device is triggered and the accumulator is reset; if the threshold is reached, the completion status is maintained and no alarm is triggered.

[0037] In an embodiment of the present invention, based on the inspection surface result output in step 6 (such as "front", "side X", etc.), the current value of the continuous inspection time accumulator corresponding to the inspection surface is retrieved from the system storage module (the accumulator value is 0 in the initial state); the detection system executes the detection process according to a fixed period, and the time step value of the current detection period is set according to the real-time requirements (usually 20ms-200ms to ensure both response speed and computing efficiency); the current value of the accumulator is added to the time step value to generate an updated accumulated time (for example: the current value is 800ms, the time step is 200ms, and it is 1000ms after update). Preset the minimum inspection time threshold (set to 1s-3s according to product inspection requirements to ensure that the inspected surface is stably displayed), compare the updated cumulative time with the threshold to determine whether the completion conditions are met.

[0038] Completion signal generation: If the updated cumulative duration reaches or exceeds the minimum detection duration threshold, the system automatically generates a completion signal for the inspected surface (such as marking it as "detected") and locks the accumulator (stops subsequent value updates) to avoid repeated counting. If the accumulated time does not reach the threshold, the updated accumulated time is stored in the system cache and used as the initial value of the accumulator for the next detection cycle, and the accumulated time continues. When the system detects that the result of the inspected surface has changed (for example, switching from "front" to "side Y"), the updated value of the original inspected surface accumulator is immediately retrieved: If the original accumulator value is less than the minimum detection time threshold, it is determined that the effective detection is not completed, and the sound and light alarm device is triggered (such as a buzzer sounding, a warning light flashing), and the accumulator value of the original inspected surface is reset to 0. If the original accumulator value has reached the threshold, its "completed" status is maintained, and no alarm is triggered. Only the initial value of the accumulator of the new inspected surface is reset to 0 and a new timing is started. Through the continuous time accumulation and threshold judgment mechanism, false completion judgments caused by instantaneous display or operational jitter are avoided, ensuring that each inspected surface is stably inspected for a sufficient period of time, thereby improving the credibility of the results. The audio-visual alarm device can promptly remind the operator when the inspected surface has not completed effective inspection, reducing the risk of missed inspections; the completion status locking mechanism prevents repeated inspections and improves process efficiency. The configurability of the time step and threshold (20ms-200ms step, 1s-3s threshold) can adapt to the inspection requirements of different products, accommodate the differences in operator operating habits, and enhance system flexibility. The accumulator's independent timing and reset logic for each surface ensures accurate status recording during multi-surface inspection, avoids timing interference between different inspected surfaces, and simplifies the complexity of system status management.

Claims

1. A method for determining the completion status of six sides of a handheld product based on cross-view image fusion, characterized in that: The method comprises: Step 1: Establish a global world coordinate system A and calibrate the three-dimensional coordinates of the operator's pupil center; establish a product body coordinate system B and calibrate the three-dimensional coordinates of the origin O and the orthogonal axial reference points X, Y, and Z; Step 2: Based on the coordinates of the reference points, a non-co-view multi-camera array is driven to synchronously capture the image of the surface to be inspected. After distortion correction through inverse projection transformation, a convolutional neural network is used to calculate the sub-pixel two-dimensional coordinates of the key points on the imaging plane. Step 3: Based on the sub-pixel 2D coordinates and the spatial coordinates of coordinate system B, the PnP algorithm is used to iteratively calculate the 3D pose data of point O and points X, Y, and Z in each camera coordinate system. Step 4: Based on the 3D pose data, the pre-calibrated rigid transformation matrix is ​​applied to map to the coordinate system A, and the spatial consistency fusion strategy is used to generate the 3D spatial consistency coordinates of point O and points X, Y, and Z in the coordinate system A; Step 5: Calculate the sight line vector from point O to the pupil center based on the pupil coordinates and the spatial consistency coordinates; calculate the product body axial vector from point O to each reference point based on the X, Y, and Z coordinates; Step 6: Based on the sight vector and the axial vector, calculate the azimuth and elevation angle feature quantities, divide the quadrants using the spatial cutting algorithm, and obtain the inspected surface result in combination with the preset spatial orientation mapping rules; Step 7: Based on the inspection results, the corresponding accumulator is activated to accumulate the continuous inspection time and compare it with the preset threshold. If the threshold is reached, a completion signal is generated; otherwise, an audible and visual alarm is triggered.

2. The method for determining the six-side completion status of a handheld product based on cross-view image fusion according to claim 1 is characterized in that: The step 2 comprises: Step 21: Generate a spatial trigger signal based on the spatial coordinates of the reference point to drive the multi-camera array in a non-common view layout to perform hardware-level synchronous exposure, thereby generating a set of original images of the surface to be inspected that are aligned in time and space. Step 22: Load the pre-calibrated camera intrinsic parameters and distortion parameters into the original image set, map the pixel coordinates to the normalized plane through inverse projection transformation, and use a nonlinear compensation function to eliminate radial and tangential distortion to obtain a geometrically corrected image set. Step 23: Define a feature search area based on the spatial coordinates of the reference points, perform multi-scale feature extraction on the geometrically corrected image set, and generate pixel-level initial coordinates of the key points; using the pixel-level initial coordinates as a reference, calculate the grayscale gradient centroid offset vector in the local neighborhood to calculate the sub-pixel two-dimensional coordinates of the key points; In step 24, a multi-view projection mapping relationship is established between the sub-pixel two-dimensional coordinates and the reference point spatial coordinates, and the reprojection error value is calculated; if the reprojection error value exceeds the tolerance threshold, the feature search area parameters are adjusted and the search is re-executed; otherwise, a sub-pixel coordinate dataset is generated.

3. The method for determining the six-side completion status of a handheld product based on cross-view image fusion according to claim 2, characterized in that: Based on the sub-pixel 2D coordinates and the spatial coordinates of coordinate system B, the PnP algorithm is used to iteratively calculate the 3D pose data of point O and points X, Y, and Z in each camera coordinate system, including: Step 31: establishing a projection correspondence relationship set between two-dimensional imaging coordinates and three-dimensional spatial coordinates based on the sub-pixel coordinate data set and the calibrated coordinate system B space coordinates; and resolving the weighted representation relationship of the spatial reference points through the virtual control point mechanism according to the three-dimensional spatial point coordinates. Step 32: construct a camera projection constraint equation based on the relationship between the coordinates of the two-dimensional imaging points and the weighted representation; solve the projection constraint equation using a matrix decomposition method to obtain the coordinates of the virtual control points in the camera coordinate system; and reconstruct the initial position of the spatial reference point in the camera coordinate system based on the relationship between the coordinates of the virtual control points and the weighted representation; Step 33: Based on the initial position of the spatial reference point, calculate the pose parameters from the product coordinate system to the camera coordinate system; perform iterative optimization of the reprojection error value based on the initial position and pose parameters until convergence, and obtain the optimized three-dimensional pose data of the camera coordinate system.

4. The method for determining the six-side completion status of a handheld product based on cross-view image fusion according to claim 3 is characterized in that: Based on the 3D pose data, a pre-calibrated rigid transformation matrix is ​​applied to map it to coordinate system A. A spatial consistency fusion strategy is used to generate the 3D spatially consistent coordinates of point O and points X, Y, and Z in coordinate system A, including: Step 41: Based on the three-dimensional pose data, the initial coordinates in the global coordinate system A are converted using a pre-calibrated rigid transformation matrix; based on the initial coordinates in the coordinate system A and the camera exposure time difference, the motion offset is calculated and the coordinate position is corrected to generate a time-synchronized coordinate set; Step 42: Calculate the fusion weight of each camera observation data based on the reprojection error value of the pose solution and the fluctuation amplitude of the time-synchronized coordinate set; perform weighted average calculation based on the fusion weight based on the time-synchronized coordinate set to generate the reference point fusion coordinates; Step 43: Calculate the distance between adjacent reference points in the fused coordinates, compare it with the original distance in the calibrated coordinate system B, eliminate the distance deviation through iterative adjustment of position offset, and output the coordinates that meet the distance consistency as the three-dimensional space consistent coordinates.

5. The method for determining the six-side completion status of a handheld product based on cross-view image fusion according to claim 4 is characterized in that: Based on the pupil coordinates and the spatial consistency coordinates of step 4, calculate the sight vector from point O to the pupil center; based on the X, Y, and Z coordinates, calculate the product body axial vector from point O to each reference point, including: Step 51: Obtain the calibrated three-dimensional coordinates of the operator's pupil center in the global world coordinate system A, and obtain the three-dimensional consistent coordinates of the origin O of the product body coordinate system and the reference points X, Y, and Z in the global world coordinate system A; Step 52: In the global world coordinate system A, perform space vector calculation based on the coordinates of the origin O and the pupil center to generate a three-dimensional sight line vector from point O to the pupil center; Step 53, in the global world coordinate system A, perform space vector calculation based on the coordinates of the origin O and the X, Y, and Z coordinates of the reference points, and generate an X-axis vector pointing from point O to the reference point X, a Y-axis vector pointing to the reference point Y, and a Z-axis vector pointing to the reference point Z.

6. The method for determining the completion status of six surfaces of a handheld product based on cross-view image fusion according to claim 5, characterized in that: Based on the line of sight vector and the axial vector, the azimuth and elevation angle features are calculated, the quadrants are divided using a spatial cutting algorithm, and the results of the inspected surface are obtained by combining the preset spatial orientation mapping rules, including: Step 61: Calculate the projection vector on the horizontal plane of the global world coordinate system A based on the sight line vector, dynamically generate a reference orientation reference axis based on the projection direction of the X-axis vector and the Y-axis vector of the product body coordinate system B on the horizontal plane, and calculate the azimuth angle feature between the projection vector and the reference orientation reference axis; Step 62: Calculate the angle between the sight line vector and the horizontal projection vector to generate a pitch angle feature; perform azimuth quadrant segmentation based on the azimuth feature, and perform pitch interval segmentation based on the pitch angle feature to form a combined spatial quadrant area; In step 63, the combined spatial quadrant area is input into a preset spatial orientation mapping rule, and the mapping rule associates the corresponding relationship between the spatial quadrant and the surface of the product hexahedron to obtain the inspection surface result.

7. The method for determining the completion status of six surfaces of a handheld product based on cross-view image fusion according to claim 1, characterized in that: Based on the inspection results, the corresponding accumulator is activated, the continuous inspection time is accumulated and compared with the preset threshold. If the threshold is reached, a completion signal is generated. Otherwise, an audible and visual alarm is triggered, including: Step 71: Based on the inspection result of the inspection surface, retrieve the current value of the continuous inspection time accumulator corresponding to the inspection surface; increase the current value of the accumulator by the time step value of the current inspection cycle to generate an updated accumulated time; Determine whether the updated cumulative duration reaches the preset minimum detection duration threshold; Step 72: If the threshold is reached, a completion signal of the inspected surface is generated and the accumulator is locked; if the threshold is not reached, the updated value of the accumulator is stored as the input reference for the next detection cycle; when a change in the inspected surface result is detected, the updated value of the original inspected surface accumulator is retrieved; if the updated value is less than the minimum detection time threshold, the sound and light alarm device is triggered and the accumulator is reset; if the threshold is reached, the completion status is maintained and no alarm is triggered.

8. The method for determining the completion status of six surfaces of a handheld product based on cross-view image fusion according to claim 7, characterized in that: The pre-calibrated camera intrinsic parameters and distortion parameters are loaded into the original image set. The pixel coordinates are mapped to the normalized plane through inverse projection transformation. The radial and tangential distortions are eliminated using a nonlinear compensation function to obtain a geometrically corrected image set, including: Step 81: Obtain the two-dimensional coordinates of the pixel coordinate system of each pixel point in the original image set and load the pre-calibrated camera intrinsic parameter matrix; construct an inverse projection mapping relationship from the pixel coordinate system to the normalized imaging plane based on the camera intrinsic parameter matrix; apply the inverse projection mapping relationship to perform coordinate transformation calculation on each pixel point to generate the corresponding normalized plane coordinates; aggregate the normalized plane coordinates of all pixel points to generate an uncorrected normalized coordinate set; Step 82 , based on the uncorrected normalized coordinate set, load the pre-calibrated radial distortion parameters and tangential distortion parameters, apply a nonlinear compensation function to calculate the distortion offset, and generate a corrected normalized coordinate set; Step 83 : Based on the corrected normalized coordinate set, reconstruct the geometric position in the pixel coordinate system through an inverse mapping relationship to generate a geometrically corrected image set.

9. The method for determining the completion status of six surfaces of a handheld product based on cross-view image fusion according to claim 8, characterized in that: Based on the uncorrected normalized coordinate set, the pre-calibrated radial distortion parameters and tangential distortion parameters are loaded, and the distortion offset is calculated using a nonlinear compensation function to generate a corrected normalized coordinate set, including: Step 91: Load the pre-calibrated radial distortion parameter sequence and tangential distortion parameter sequence, and obtain an uncorrected normalized coordinate set; for each coordinate point in the normalized coordinate set, calculate the square of the radial distance from the origin of the normalized plane; and calculate the radial distortion scaling factor based on the radial distortion parameter sequence and the square of the radial distance. Step 92: Multiply the radial distortion scaling factor by the coordinate value of the original coordinate point to generate a radial distortion offset component; calculate the coordinate product term and the radius square term based on the tangential distortion parameter sequence and the coordinate value of the original coordinate point; and sum the coordinate product term and the radius square term weighted by the tangential distortion parameter to generate a tangential distortion offset component. Step 93 , vector superposition of the radial distortion offset component and the tangential distortion offset component is performed to generate a total distortion offset; the total distortion offset is subtracted from the original uncorrected coordinates to generate corrected normalized coordinates; and all corrected coordinate points are aggregated to generate a corrected normalized coordinate set.

10. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

Images