Precise workpiece positioning method and system based on three-dimensional vision

Abstract

This invention provides a precision workpiece positioning method and system based on 3D vision. It acquires a set of 3D visual data of a scene containing the target workpiece, performs multimodal data fusion processing on the scene's 3D visual data set to generate a fused data set with spatially aligned coordinates, extracts a set of workpiece feature identifiers from the fused data set, and performs feature matching and association processing between the workpiece feature identifier set and a pre-stored standard workpiece feature template to generate a set of feature matching correspondences. Based on the set of feature matching correspondences, it calculates the 3D positioning coordinates of the target workpiece in the scene. The 3D positioning coordinates include the workpiece's centroid coordinates and spatial attitude angle parameters. This invention improves the accuracy of precision workpiece machining and ensures the stability of the positioning process.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, and in particular to a precision workpiece positioning method and system based on three-dimensional vision. Background Technology

[0002] With the development of 3D vision technology, precision workpiece positioning technology has been proposed. This technology determines the position and orientation of a workpiece in 3D space by analyzing its visual data, and is widely used in intelligent manufacturing, precision assembly, and other scenarios requiring high-precision positioning. Currently, common precision workpiece positioning technologies typically acquire visual data, extract features such as surface key points or only contour lines, and then perform matching and positioning calculations based on these features. Existing technologies often face the problem of incomplete feature extraction, leading to insufficient accuracy and stability of the positioning results. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and system for precision workpiece positioning based on three-dimensional vision. The technical solution of the embodiments of the present invention is implemented as follows:

[0004] On one hand, embodiments of the present invention provide a precision workpiece positioning method based on three-dimensional vision. The method includes: acquiring a scene three-dimensional vision data set containing a target workpiece, the scene three-dimensional vision data set consisting of workpiece surface point cloud data synchronously acquired from multiple perspectives and two-dimensional image data from corresponding perspectives; performing multimodal data fusion processing on the scene three-dimensional vision data set to generate a fused data group with spatial coordinate alignment, the fused data group containing a mapping relationship between point cloud coordinates and two-dimensional image pixel coordinates; extracting a workpiece feature identifier set from the fused data group, the workpiece feature identifier set containing key feature point location information and continuous contour line feature description on the workpiece surface; performing feature matching association processing on the workpiece feature identifier set and a pre-stored standard workpiece feature template to generate a feature matching correspondence set, the feature matching correspondence set containing positional correspondence of key feature points and shape matching relationship of contour lines; calculating the three-dimensional positioning coordinate information of the target workpiece in the scene based on the feature matching correspondence set, the three-dimensional positioning coordinate information containing workpiece centroid coordinates and spatial attitude angle parameters.

[0005] On the other hand, embodiments of the present invention provide a computer system including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in the above-described method.

[0006] This invention presents a precision workpiece positioning method based on 3D vision. It simultaneously acquires point cloud data and 2D image data from multiple perspectives to generate spatially aligned fused data sets, integrating the 3D coordinate information of the point cloud with the texture and color information of the image. This avoids the limitations of simple modal data in spatial information or detail representation. It extracts dual feature identifiers: key feature point position information and continuous contour line feature descriptions. These provide local positional constraints through key feature points and global shape constraints through continuous contour lines, compensating for deficiencies in positioning constraints. Through composite feature association of the positional correspondence of key feature points and the shape matching relationship of contour lines, it achieves synergy between local position verification and global shape verification, reducing the risk of misjudgment due to insufficient feature extraction. Finally, it calculates 3D positioning coordinates based on the dual feature correspondence, combining local coordinate transformation constraints and global shape consistency constraints. This ensures that the positioning result reflects both the positional relationships of discrete features on the workpiece surface and conforms to the shape rules of the overall workpiece contour, thereby improving the accuracy and reliability of precision workpiece positioning. Simultaneously, the fusion processing of multimodal data and the composite matching of dual features, through information complementarity and constraint superposition, enhance the method's adaptability to complex surface workpieces or partially occluded scenarios, ensuring the stability of the positioning process. Attached Figure Description

[0007] Figure 1 This is a schematic diagram illustrating the implementation process of a precision workpiece positioning method based on three-dimensional vision, provided in an embodiment of the present invention.

[0008] Figure 2 This is a schematic diagram of the hardware entity of a computer system provided in an embodiment of the present invention. Detailed Implementation

[0009] This invention provides a precision workpiece positioning method based on three-dimensional vision, which can be executed by a computer system processor. The computer system can refer to the computer system in the control system of a precision workpiece processing production line. The precision workpiece is, for example, the casing of a mobile phone camera module, and the processing can involve applying a film to it.

[0010] Figure 1 This is a schematic diagram illustrating the implementation process of a precision workpiece positioning method based on three-dimensional vision, as provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0011] Step S100: Obtain a scene 3D vision data set containing the target workpiece. The scene 3D vision data set consists of workpiece surface point cloud data acquired synchronously from multiple perspectives and corresponding 2D image data.

[0012] The scene's 3D vision dataset serves as the foundation for subsequent precision workpiece positioning, integrating point cloud data and 2D image data of the workpiece surface from multiple perspectives. Point cloud data consists of a set of points representing the spatial positions of an object's surface, each point possessing 3D spatial coordinate information that accurately reflects the workpiece's surface geometry. 2D image data contains texture and color information of the workpiece surface, providing rich appearance features. Simultaneous acquisition from multiple perspectives ensures comprehensive and accurate acquisition of workpiece information, avoiding issues of occlusion or missing information due to a single perspective.

[0013] For example, in a scenario where a screen protector is automatically positioned for a mobile phone camera, the target workpiece is the phone's camera module. To acquire a set of 3D visual data containing this target workpiece, specialized acquisition equipment and techniques are required. For instance, a high-precision 3D vision acquisition system might be used, which could integrate a LiDAR sensor and an RGB camera. In practice, the phone is placed on a workbench, and the 3D vision acquisition equipment is controlled to collect data from multiple perspectives around the phone's camera module.

[0014] As one implementation method, step S100 can be specifically implemented as the following steps S110~S150:

[0015] Step S110: Control the three-dimensional vision acquisition device to acquire data at multiple preset observation positions around the workpiece, and set fixed device pose parameters for each observation position.

[0016] A 3D vision acquisition device is a hardware device used to acquire 3D information and 2D image information of a workpiece. It can observe and acquire data from the workpiece from different angles and positions. Multiple preset observation positions are planned in advance based on the shape, size, and features of the workpiece, ensuring comprehensive and accurate acquisition of workpiece information. Device pose parameters describe the device's position and orientation in space, including the device's origin coordinates and optical axis direction vector. Setting fixed device pose parameters ensures consistent device state during each data acquisition, improving data accuracy and repeatability.

[0017] As one implementation method, step S110 can be specifically implemented as the following steps S111~S116:

[0018] Step S111: Perform pre-scanning processing on the workpiece surface to generate preliminary point cloud data containing surface feature distribution, which includes texture complexity and curvature change rate information.

[0019] Pre-scanning is a rapid scan of the workpiece surface performed before formal data acquisition. Its purpose is to obtain approximate feature information of the workpiece surface. Preliminary point cloud data is obtained through pre-scanning and contains basic geometric shape and feature distribution information of the workpiece surface. Texture complexity reflects the richness and complexity of the workpiece surface texture, while the rate of curvature change indicates how quickly the curvature of the workpiece surface changes. These two parameters help identify feature-rich and feature-sparse regions on the workpiece surface.

[0020] For example, in a scenario where a screen protector is automatically positioned for a mobile phone camera, a low-resolution LiDAR sensor can be used to pre-scan the surface of the camera module. For instance, the LiDAR sensor is configured to quickly scan the camera module surface with low precision, acquiring a series of point cloud data. This point cloud data is then processed and analyzed to calculate the texture complexity and rate of curvature change for each point. The rate of curvature change can be calculated by statistically analyzing the distance and angle changes between adjacent points in the point cloud data, and the texture complexity can be evaluated by analyzing the local variance of the corresponding grayscale image. For the lens portion of the mobile phone camera, due to its smooth surface, the rate of curvature change is small, and the texture complexity is relatively low; while for the markings and fixed structures around the lens, the rate of curvature change is large, and the texture complexity is also high.

[0021] Step S112: Construct an observation location planning model based on surface feature distribution information. The observation location planning model aims to optimize the coverage integrity of feature-rich areas and the redundancy control of feature-sparse areas.

[0022] The observation location planning model is a mathematical model used to determine the optimal observation location. It plans the observation location of the acquisition equipment based on the feature distribution information of the workpiece surface. Feature-rich regions refer to areas on the workpiece surface with high texture complexity and a large rate of curvature change. These regions contain more feature information and are crucial for accurate workpiece identification and positioning; therefore, it is necessary to ensure that these regions are completely covered. Feature-sparse regions refer to areas with low texture complexity and a small rate of curvature change. To avoid unnecessary data redundancy, observations in these regions need to be reasonably controlled to reduce duplicate acquisitions.

[0023] For example, an observation location planning model can be constructed based on the surface feature distribution information obtained from pre-scanning. Optimization algorithms such as genetic algorithms or particle swarm optimization can be used to construct the model. Taking a genetic algorithm as an example, firstly, the coordinates and angles of the observation locations are treated as genes, and each possible combination of observation locations is treated as an individual, forming a population. Then, a fitness function is defined, with the optimization objective being the complete coverage of feature-rich regions and the control of redundancy in feature-sparse regions. Through continuous selection, crossover, and mutation operations, the population is iteratively updated until the optimal combination of observation locations is found. For example, for feature-rich regions around the lens of a mobile phone camera module, the model will plan more observation locations to ensure that the feature information of these regions can be completely collected; while for feature-sparse regions on the lens surface, the number of observation locations will be appropriately reduced to avoid data redundancy.

[0024] Step S113: Dynamically determine the set of target observation positions around the workpiece through the observation position planning model. The density of observation points in feature-rich areas of the target observation position set is higher than that in feature-sparse areas.

[0025] The target observation location set is the final set of observation locations determined by the observation location planning model. It ensures that when collecting data, more intensive observations are made on feature-rich areas of the workpiece surface, while relatively sparse observations are made on feature-sparse areas, thereby reducing unnecessary data collection while ensuring data integrity.

[0026] In the scenario of automatic positioning and film application for mobile phone cameras, the surface feature distribution information obtained from pre-scanning is input into the observation position planning model. Through model calculation and optimization, the set of target observation positions is dynamically determined. For example, for feature-rich areas around the lens of the mobile phone camera module, 5-10 observation positions are set, while for sparsely featured areas on the lens surface, 2-3 observation positions are set. This ensures that key feature information of the workpiece surface can be obtained more effectively during data collection, improving the accuracy of positioning.

[0027] Step S114: At each target observation position, adjust the lens orientation of the 3D vision acquisition device through the pose calibration module so that the lens optical axis and the normal vector of the corresponding area of ​​the workpiece surface maintain a preset angle relationship.

[0028] The pose calibration module is used to precisely adjust the pose of the 3D vision acquisition device. It can accurately adjust the lens orientation of the device according to the requirements of the target observation position. The normal vector is a vector perpendicular to a point on the workpiece surface, representing the surface direction at that point. The preset angle relationship is the angle between the lens optical axis and the workpiece surface normal vector, pre-set according to actual needs. Maintaining this angle relationship ensures that the acquired images and point cloud data have optimal quality and accuracy.

[0029] In the scenario of automatic positioning and film application using a mobile phone camera, the pose calibration module begins operation after the 3D vision acquisition device moves to each target observation position. This module utilizes a high-precision angle sensor and a motor control system to adjust the lens orientation. For example, the angle sensor monitors in real time the angle between the lens optical axis and the normal vector of the corresponding area on the workpiece surface. Then, based on a preset angle relationship, it controls the motor to drive the lens to rotate and tilt until the preset angle requirement is achieved.

[0030] Step S115: Record the device spatial coordinates and lens orientation parameters of each target observation position as fixed device pose parameters. The device pose parameters include the device origin coordinates and the optical axis direction vector.

[0031] Equipment spatial coordinates refer to the equipment's position coordinates in three-dimensional space, while lens orientation parameters describe the lens's direction and angle. Recording these parameters as fixed equipment pose parameters ensures accurate determination of the equipment's status at each observation location during subsequent data acquisition and processing, guaranteeing data consistency and traceability. The equipment origin coordinates are the coordinates of the origin of the equipment coordinate system in the global coordinate system, while the optical axis direction vector represents the direction of the lens's optical axis.

[0032] In the scenario of automatic positioning and screen protector application for mobile phone cameras, a high-precision positioning system and angle sensors are used to record the device's spatial coordinates and lens orientation parameters at each target observation position. For example, a laser tracker or GPS positioning system is used to obtain the device's spatial coordinates, and an electronic compass or gyroscope is used to obtain the lens orientation parameters. These parameters are recorded and stored in a computer database as a reference for subsequent data acquisition and processing. For each target observation position, there is a unique set of device pose parameters that accurately describe the device's state at that location.

[0033] Step S116: Based on the device pose parameters, control the 3D vision acquisition device to synchronously acquire point cloud data and 2D image data at each observation position to form a scene 3D vision data set covering the feature distribution of the workpiece surface.

[0034] Controlling a 3D vision acquisition device based on its pose parameters means precisely controlling the acquisition device to collect data at each observation position according to previously recorded device pose parameters. Synchronous acquisition refers to simultaneously acquiring point cloud data and 2D image data, ensuring accurate correspondence between the two types of data. A scene-based 3D vision dataset covering the feature distribution of the workpiece surface refers to acquired data that comprehensively and accurately reflects the feature distribution of the workpiece surface, including both feature-rich and feature-sparse regions.

[0035] In the scenario of automatic screen protector placement for mobile phone cameras, based on recorded device pose parameters, a 3D vision acquisition device is controlled to synchronously acquire data at each target observation position. For example, at the observation position directly above the lens, the device adjusts its position and orientation according to preset pose parameters, and then simultaneously activates the LiDAR sensor and RGB camera to acquire data. The LiDAR sensor acquires point cloud data of the mobile phone camera module surface at that position, and the RGB camera acquires the corresponding 2D image data. This process is repeated, acquiring data at each target observation position. Finally, all the acquired point cloud data and 2D image data are integrated to form a scene 3D vision data set covering the feature distribution of the mobile phone camera module surface.

[0036] Step S120: At each observation position, point cloud data of the workpiece surface is acquired through a lidar sensor. The point cloud data contains the three-dimensional spatial coordinate information of each point.

[0037] LiDAR sensors can accurately measure the three-dimensional spatial coordinates of every point on the surface of an object, thereby generating point cloud data. Each point in the point cloud data represents a location on the workpiece surface, and its three-dimensional spatial coordinate information can accurately describe the geometry of the workpiece.

[0038] In the scenario of automatic screen protector placement for mobile phone cameras, at each observation position, a LiDAR sensor emits a laser beam towards the surface of the phone's camera module. For example, the LiDAR sensor emits laser pulses at a certain frequency. When the laser pulses encounter the surface of the camera module, they are reflected back and received by the sensor. By measuring the time difference between the emission and reception of the laser pulses, combined with the laser's propagation speed, the distance between each reflection point and the sensor can be calculated. Simultaneously, based on the sensor's position and angle information, this distance information is converted into three-dimensional spatial coordinates. For the lens portion of the mobile phone camera, the LiDAR sensor can acquire a series of precise point cloud data, which can accurately depict the circular shape and surface curvature of the lens.

[0039] Step S130: At the same observation position, two-dimensional image data of the corresponding viewpoint are acquired synchronously by an RGB camera. The two-dimensional image data contains the texture and color information of the workpiece surface.

[0040] RGB cameras record image color information through three color channels (red, green, and blue). Simultaneous acquisition of 2D image data from the same observation location ensures viewpoint consistency between point cloud data and 2D image data, facilitating subsequent multimodal data fusion processing. Texture and color information in 2D image data can provide more visual features, aiding in more accurate workpiece identification and positioning.

[0041] In the scenario of automatic positioning and film application for mobile phone cameras, at each observation position, while the LiDAR sensor acquires point cloud data, the RGB camera simultaneously acquires 2D image data from the corresponding viewpoint. For example, the lens of the RGB camera maintains the same orientation and position as the lens of the LiDAR sensor to ensure that the acquired 2D image data and point cloud data have the same viewpoint. For the surface of the mobile phone camera module, the RGB camera can capture information such as the glass gloss of the lens, the color and texture of surrounding markings. This texture and color information can be combined with the geometric information in the point cloud data to provide richer features for subsequent workpiece positioning.

[0042] Step S140: Record the device pose parameters at each observation position as a spatial reference benchmark. The spatial reference benchmark includes the device's position and orientation information relative to the scene's fixed coordinate system.

[0043] A spatial reference datum is a standard used to determine the position and orientation of data in three-dimensional space. It records the device pose parameters at each observation location, including the device's position and orientation relative to the scene's fixed coordinate system. The scene's fixed coordinate system is a predefined global coordinate system, and all data can be transformed into this coordinate system through the spatial reference datum for unified processing and analysis. In the scenario of automatic positioning and screen protector application for a mobile phone camera, the device pose parameters at each observation location are recorded as the spatial reference datum. High-precision positioning systems and angle sensors can be used to obtain the device's position and orientation information relative to the scene's fixed coordinate system. For example, the phone is placed on a fixed workbench, and a scene's fixed coordinate system is established with one corner of the workbench as the origin. At each observation location, the device's origin coordinates in the scene's fixed coordinate system are measured using a laser tracker or GPS positioning system, and the angle of the device's optical axis direction vector relative to the scene's fixed coordinate system is measured using an electronic compass or gyroscope. This information is recorded as the spatial reference datum for subsequent data storage and processing.

[0044] Step S150: Store the point cloud data and two-dimensional image data collected at each observation location according to the corresponding spatial reference benchmark to form a scene three-dimensional visual data set composed of multiple sets of point cloud-image data pairs.

[0045] Storing point cloud data and 2D image data collected at each observation location according to their corresponding spatial reference datum ensures spatial consistency and traceability of the data. The point cloud data and 2D image data at each observation location constitute a point cloud-image data pair, which contains geometric and texture color information of the workpiece surface from the same viewpoint. Integrating all point cloud-image data pairs forms a 3D visual data set for the scene. In the scenario of automatic film application using a mobile phone camera, the point cloud data and 2D image data collected at each observation location are stored according to their corresponding spatial reference datum.

[0046] Step S200: Perform multimodal data fusion processing on the scene 3D visual data set to generate a fused data set with spatial coordinate alignment. The fused data set contains the mapping relationship between point cloud coordinates and 2D image pixel coordinates.

[0047] Multimodal data fusion processing integrates and processes different types of data (such as point cloud data and 2D image data) to obtain more comprehensive and accurate information. Spatial coordinate alignment refers to unifying point cloud data and 2D image data from different observation locations into the same spatial coordinate system, making them spatially consistent. The fused data set is the dataset obtained after multimodal data fusion processing, containing the mapping relationship between point cloud coordinates and 2D image pixel coordinates. This mapping relationship can convert the 3D coordinates in the point cloud data to pixel coordinates in the 2D image, and vice versa. In the scenario of automatic positioning and screen protector application using a mobile phone camera, multimodal data fusion processing is performed on the collected 3D visual data set of the scene. First, it is necessary to solve the spatial alignment problem of data from different observation locations. Since the acquisition devices at different observation locations may have different poses, the acquired data will differ in spatial coordinate systems. Through multimodal data fusion processing, these data can be unified into a common coordinate system. Then, a mapping relationship between point cloud coordinates and 2D image pixel coordinates is established to help associate the geometric information in the point cloud data with the texture and color information in the 2D image data in subsequent processing. For example, for a point cloud data point of a mobile phone camera module, its corresponding pixel in a two-dimensional image can be found through mapping relationships.

[0048] As one implementation method, step S200 can be specifically implemented as the following steps S210~S250:

[0049] Step S210: Select any point cloud-image data pair from the scene 3D visual data set as the reference data set, and extract the feature points of the 2D image of the reference data set.

[0050] The baseline data set is a representative set of data selected from the scene's 3D visual data collection. It will serve as a reference standard for subsequent data fusion and processing. Feature points are points in a 2D image that have significant features, such as corner points and edge points. These points contain important structural information of the image and are crucial for image matching and coordinate alignment.

[0051] In the scenario of automatic screen protector placement using a mobile phone camera, a set of point cloud-image data pairs is randomly selected from the acquired 3D visual data set of the scene as the baseline data set. For example, the point cloud-image data pair at the observation position directly above the lens is selected. Then, feature extraction algorithms are used to extract feature points from the 2D images of the baseline data set. Algorithms such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Robust Feature Transform) can be used. Taking the SIFT algorithm as an example, a Gaussian difference pyramid is first constructed on the 2D image, and then extreme points are found in each layer of the pyramid. Next, these extreme points are precisely located and filtered to remove unstable points. Finally, the feature descriptor of each feature point is calculated for subsequent feature matching. For locations such as the mobile phone camera logo and edges, more feature points are extracted, and these feature points can represent the key structures of the image.

[0052] Step S220: Use a feature matching algorithm to perform feature matching between the two-dimensional image data from other observation locations and the two-dimensional image data from the reference data set to obtain the transformation relationship between images from different observation locations.

[0053] Feature matching algorithms are used to find matching feature points in different images. They determine which feature points correspond by comparing their feature descriptors. Through feature matching, correspondences between images viewed from different locations can be found, and their transformation relationships can be calculated. The transformation relationship describes how a point in one image is transformed to its corresponding position in another image, expressed using transformation parameters such as rotation, translation, and scaling.

[0054] In the scenario of automatic screen protector placement using a mobile phone camera, a feature matching algorithm is used to perform feature matching between 2D image data from other observation positions and the 2D image data of the baseline data set. The RANSAC (Random Sample Consensus) algorithm combined with SIFT feature matching can be employed. The specific steps are as follows: First, feature point extraction and feature descriptor calculation are performed on the 2D image data from other observation positions, using the same method as the baseline data set. Then, preliminary feature matching is performed using SIFT feature descriptors to find possible matching point pairs. Next, the RANSAC algorithm is used to filter these matching point pairs, removing incorrect matches. The RANSAC algorithm randomly selects a set of matching point pairs, calculates the transformation relationship between them, and then verifies other matching point pairs based on this transformation relationship, retaining those that meet the criteria. Finally, based on the filtered matching point pairs, the transformation relationship between images from different observation positions is calculated. For example, for 2D image data from an observation position diagonally above the lens, the rotation, translation, and scaling parameters relative to the baseline data set can be calculated using feature matching and the RANSAC algorithm, thus obtaining the transformation relationship between them.

[0055] As one implementation method, step S220 can be specifically implemented as the following steps S221~S227:

[0056] Step S221: Extract feature points from the two-dimensional image data of the baseline data set, and simultaneously obtain the three-dimensional coordinates of each feature point in the corresponding point cloud data as feature space anchor points.

[0057] Feature point extraction involves using feature extraction algorithms to identify points with significant features from a two-dimensional image. Feature space anchor points are the corresponding points of feature points in three-dimensional space. By associating feature points with point cloud data, the three-dimensional coordinates of each feature point can be obtained. These three-dimensional coordinates will serve as reference points for subsequent spatial matching and transformation.

[0058] In the scenario of automatic screen protector placement using a mobile phone camera, feature points are extracted from the 2D image data of the baseline dataset. The SIFT algorithm can be used for feature point extraction, as it can find scale-invariant and rotation-invariant feature points in the image. For each extracted feature point, its 3D coordinates in the corresponding point cloud data are found using the previously established mapping relationship between point cloud coordinates and 2D image pixel coordinates. For example, for a feature point on the lens edge in the baseline dataset's 2D image, its corresponding 3D coordinates can be found in the point cloud data through the mapping relationship; these 3D coordinates are the feature space anchor point of that feature point.

[0059] Step S222: Perform the same feature point extraction and feature space anchor point acquisition operations on the two-dimensional image data of other observation locations to generate a feature point-space anchor point association set for the corresponding observation location.

[0060] Performing the same operation on the 2D image data from other observation locations ensures consistency and comparability of data across all locations. The feature point-spatial anchor association set is a collection recording the relationships between feature points at each observation location and their corresponding feature spatial anchors; it serves as the foundation for subsequent feature matching and spatial transformation.

[0061] In the scenario of automatic screen protector placement using a mobile phone camera, the SIFT algorithm is also used to extract feature points from the 2D image data of observation positions other than the baseline data set. Then, based on the mapping relationship between the corresponding point cloud and image data pairs, the feature space anchor point of each feature point is obtained. The feature points of each observation position are associated with their corresponding feature space anchor points to generate a feature point-space anchor point association set. For example, for the 2D image data of the observation position on the side of the lens, after extracting a series of feature points, their 3D coordinates in the point cloud data are found through the mapping relationship, forming a feature point-space anchor point association set for that observation position.

[0062] Step S223: Calculate the image feature descriptor similarity between the feature points of the baseline data group and the feature points of other observation locations, and generate an initial similarity matching matrix.

[0063] Image feature descriptors are vectors used to describe the features of feature points, containing local image information about those points. Similarity refers to the degree of similarity between two feature descriptors; calculating the similarity determines whether two feature points match. The initial similarity matching matrix is ​​a matrix that records the similarity between feature points in the baseline dataset and feature points at other observation locations, providing preliminary information for subsequent feature matching.

[0064] In the scenario of automatic screen protector placement using a mobile phone camera, the similarity of image feature descriptors between feature points in the baseline data set and feature points at other observation locations is calculated. Methods such as Euclidean distance or cosine similarity can be used. Taking Euclidean distance as an example, a smaller distance indicates a higher similarity. The similarity calculation results of all feature points in the baseline data set and feature points at other observation locations are stored in a matrix to form an initial similarity matching matrix.

[0065] Step S224: Based on the three-dimensional coordinate information of the feature space anchor points, calculate the spatial Euclidean distance between the feature points of the benchmark data group and the feature points of other observation locations, and generate a spatial distance matching matrix.

[0066] Spatial Euclidean distance refers to the distance between two feature space anchor points in three-dimensional space, reflecting the relative positional relationship of feature points in space. The spatial distance matching matrix is ​​a matrix that records the spatial Euclidean distances between feature points in the baseline data set and feature points in other observation locations. It is used together with the initial similarity matching matrix for subsequent comprehensive matching.

[0067] In the scenario of automatic screen protector placement using a mobile phone camera, the spatial Euclidean distance between feature points in the baseline data set and feature points at other observation locations is calculated based on the 3D coordinate information of feature space anchor points. For the feature space anchor point of a feature point in the baseline data set and the feature space anchor points of a feature point at another observation location, the distance between them is calculated using the Euclidean distance formula based on their 3D coordinates (x1, y1, z1) and (x2, y2, z2). The spatial Euclidean distance calculation results of all feature points in the baseline data set and feature points in other observation locations are stored in a matrix to form a spatial distance matching matrix.

[0068] Step S225: Perform weighted fusion processing on the initial similarity matching matrix and the spatial distance matching matrix to generate a comprehensive matching score matrix. The weight parameters of the weighted fusion are dynamically adjusted according to the texture complexity of the region where the feature points are located.

[0069] The weighting parameter controls the proportion of the initial similarity matching matrix and the spatial distance matching matrix during the fusion process. It is dynamically adjusted based on the texture complexity of the region where the feature points are located to adapt to the feature matching requirements of different regions. In the scenario of automatic positioning and screen protector application by a mobile phone camera, the initial similarity matching matrix and the spatial distance matching matrix are weighted and fused. First, the weighting parameter is determined based on the texture complexity of the region where the feature points are located. For regions with high texture complexity, the similarity of image features may be more important, so the initial similarity matching matrix is ​​given a higher weight; for regions with low texture complexity, the consistency of spatial location may be more important, so the spatial distance matching matrix is ​​given a higher weight. Then, the initial similarity matching matrix and the spatial distance matching matrix are weighted and summed according to their respective weights to obtain the comprehensive matching score matrix.

[0070] Step S226: Select feature point pairs with a comprehensive matching score higher than a preset threshold as candidate matching pairs. The candidate matching pairs simultaneously meet the requirements of image feature similarity and spatial proximity.

[0071] The preset threshold is a pre-defined score standard used to filter feature point pairs with high matching scores. Candidate matching pairs refer to feature point pairs with a comprehensive matching score higher than the preset threshold. These feature point pairs simultaneously meet the requirements of image feature similarity and spatial proximity, and they form the basis for subsequent topological consistency verification.

[0072] Step S227: Perform topological consistency verification on the candidate matching pairs to ensure that the relative positional relationship between feature points in the baseline data group is consistent with the relative positional relationship between feature points in other observation locations. Retain the candidate matching pairs that pass the topological consistency verification as valid matching pairs, and solve the transformation relationship parameters between images at different observation locations based on the coordinate information of the valid matching pairs.

[0073] Topological consistency verification is a process of further screening and validating candidate matching pairs. It determines the validity of a matching pair by comparing the relative positional relationships between feature points in the baseline data set with those between feature points in other observation locations. Valid matching pairs are candidate matching pairs that have passed topological consistency verification. They have high matching accuracy, and the transformation parameters between images at different observation locations can be accurately solved based on the coordinate information of these matching pairs.

[0074] In the scenario of automatic screen protector placement using a mobile phone camera, topological consistency verification is performed on candidate matching pairs. A graph-based matching method can be used, treating feature points in the baseline data set and feature points at other observation locations as nodes in a graph, and the relative positions of these feature points as edges. By comparing the structures and edge connections of the two graphs, it is determined whether the candidate matching pairs satisfy topological consistency. For example, for the three feature points A, B, and C in the baseline data set, their relative distances and angular relationships constitute the structure of one graph; for the three corresponding candidate matching feature points A', B', and C' at other observation locations, their relative distances and angular relationships constitute the structure of another graph. If the structures of the two graphs are similar, i.e., the relative positions of the feature points are consistent, then this set of candidate matching pairs is considered to have passed the topological consistency verification. The verified candidate matching pairs are retained as valid matching pairs. Then, methods such as least squares or singular value decomposition are used to solve for the transformation parameters between images at different observation locations, including rotation, translation, and scaling parameters, based on the coordinate information of these valid matching pairs.

[0075] Step S230: Based on the transformation relationship and the spatial reference datum of each observation location, calculate the coordinate transformation parameters of the point cloud data at other observation locations relative to the reference data set point cloud data.

[0076] Coordinate transformation parameters are used to transform point cloud data from other observation locations into the spatial coordinate system of the reference data set's point cloud data. They are calculated based on the transformation relationship between images from different observation locations and the spatial reference datum for each observation location. Coordinate transformation enables spatial alignment of point cloud data from different observation locations. In the scenario of automatic screen protector application for a mobile phone camera, based on the transformation relationship between images from different observation locations obtained in step S220 and the spatial reference datum for each observation location, coordinate transformation parameters of point cloud data from other observation locations relative to the reference data set's point cloud data are calculated. First, the transformation relationship is represented as a transformation matrix, which includes transformation information such as rotation, translation, and scaling. Then, combined with the spatial reference datum for each observation location, the coordinate system of the point cloud data from other observation locations is transformed into the coordinate system of the reference data set's point cloud data. For example, for point cloud data from the observation location on the side of the lens, a coordinate transformation matrix is ​​calculated based on its transformation relationship matrix with the reference data set and its own spatial reference datum. Multiplying the coordinates of each point in the point cloud data at this observation location by this coordinate transformation matrix transforms it into the spatial coordinate system of the reference data set's point cloud data.

[0077] Step S240: Use coordinate transformation parameters to transform the point cloud data from other observation locations to the spatial coordinate system of the reference data set, and then stitch them together with the point cloud data of the reference data set to generate global point cloud data.

[0078] Transforming point cloud data from other observation locations using coordinate transformation parameters ensures spatial consistency, enabling stitching and fusion. Global point cloud data, obtained by stitching together point cloud data from all observation locations, contains complete geometric information about the workpiece surface.

[0079] In the scenario of automatic positioning and screen protector application for the mobile phone camera, the coordinate transformation parameters calculated in step S230 are used to transform the point cloud data from other observation positions to the spatial coordinate system of the reference data set. For example, for the point cloud data at the observation position diagonally above the lens, the coordinates of each point are transformed to the coordinate system of the reference data set using a coordinate transformation matrix. Then, the transformed point cloud data is stitched together with the point cloud data from the reference data set. For points in overlapping areas, the averaging method or the nearest neighbor method can be used to eliminate data redundancy and inconsistency. Finally, a global point cloud data containing complete geometric information of the mobile phone camera module surface is generated.

[0080] Step S250: Perform projection transformation processing on the global point cloud data and the two-dimensional image data at each observation location, project the three-dimensional coordinates of the point cloud onto the pixel coordinate system of the corresponding two-dimensional image through camera parameters, establish the mapping relationship between the point cloud coordinates and the image pixel coordinates, and form a fused data group with spatial coordinate alignment.

[0081] Projection transformation is the process of converting point cloud coordinates in three-dimensional space to pixel coordinates in a two-dimensional image. It is achieved through parameters such as the camera's intrinsic and extrinsic parameters. Establishing a mapping relationship between point cloud coordinates and image pixel coordinates allows us to associate the geometric information in the point cloud data with the texture and color information in the two-dimensional image data, forming a fused data set with spatially aligned coordinates.

[0082] In the scenario of automatic screen protector placement using a mobile phone camera, projection transformation is performed on the global point cloud data and the 2D image data at each observation position. First, the camera's intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as rotation matrix and translation vector) are obtained. Then, based on these parameters, the 3D coordinates of each point in the global point cloud data are projected onto the pixel coordinate system of the corresponding 2D image. A pinhole camera model can be used for projection transformation; the projection formula for this model is: Where (u,v) are pixel coordinates, (X,Y,Z) are the 3D coordinates of the point cloud, f is the focal length, and (c x ,c y The coordinates are the principal points. This projection transformation establishes a mapping between the point cloud coordinates and the image pixel coordinates. After processing all the point cloud data and 2D image data in this way, a spatially aligned fused data set is formed. This fused data set contains complete geometric and texture color information of the phone camera module surface, and they are spatially aligned.

[0083] Step S300: Extract the workpiece feature identifier set from the fused data set. The workpiece feature identifier set contains the location information of key feature points on the workpiece surface and the feature description of continuous contour lines.

[0084] A workpiece feature identifier set is a group of information used to describe the surface features of a workpiece. It includes the location information of key feature points and the feature description of continuous contour lines. This information is crucial for accurately identifying and locating the workpiece. Key feature points are points on the workpiece surface with significant features, such as corners and endpoints, while continuous contour lines are the boundary lines of the workpiece surface, describing the workpiece's external shape.

[0085] In the scenario of automatic positioning and film application for mobile phone cameras, a set of workpiece feature identifiers is extracted from the fused data set. The fused data set contains point cloud data and 2D image data of the surface of the mobile phone camera module, and they are spatially aligned. By processing and analyzing this data, the positional information of key feature points and the feature description of continuous contour lines can be extracted. For example, key feature points can be identified for the lens edge and markings of the mobile phone camera; continuous contour lines can be extracted for the overall shape of the camera module.

[0086] As one implementation method, step S300 can be specifically implemented as the following steps S310~S370:

[0087] Step S310: Perform noise filtering on the global point cloud data in the fused data group to eliminate discrete noise points in the point cloud data.

[0088] Noise filtering is the process of denoising global point cloud data. Discrete noise points are isolated points in point cloud data caused by factors such as measurement errors or environmental interference. These points will affect the subsequent feature extraction and analysis results, so they need to be eliminated.

[0089] In scenarios where mobile phone cameras automatically locate and apply screen protectors, noise filtering is performed on the global point cloud data in the fused data set. Methods such as statistical filtering or radius filtering can be used. Taking statistical filtering as an example, first, the average distance to the neighboring points of each point is calculated. Then, the average distance and standard deviation are used to determine whether the point is a noise point. If the average distance to the neighboring points of a point is much greater than the global average distance, the point is considered a noise point and is removed from the point cloud data. For some isolated outliers in the mobile phone camera point cloud data, statistical filtering can effectively eliminate them, making the point cloud data smoother and more accurate.

[0090] Step S320: Identify salient feature regions in the smoothed point cloud data using a feature detection algorithm, and use the center point of the region as a candidate key feature point.

[0091] Feature detection algorithms are used to find regions with salient features in point cloud data. A salient region is a region in point cloud data that has unique geometric features, such as corners and edges. Candidate key feature points are points selected from these salient regions; they will be used for further screening and verification.

[0092] In scenarios where screen protectors are automatically positioned for mobile phone camera applications, feature detection algorithms are used to identify salient regions in smoothed point cloud data. Algorithms such as ISS (Intrinsic Shape Signature) or Harris3D can be employed. Taking the ISS algorithm as an example, the local surface features of each point in the point cloud data, such as curvature and normals, are first calculated. Then, regions with salient features are selected based on these features. For each salient region, the coordinates of its center point are calculated, and this center point is used as a candidate key feature point. For locations such as the lens edge and logos of the mobile phone camera, multiple salient regions will be identified, and the center point of each region will become a candidate key feature point.

[0093] Step S330: Perform multi-view repeatability verification on candidate key feature points, retain stable key feature points that can be detected from multiple observation views, and form a set of key feature point location information.

[0094] Multi-view repeatability verification is a process of further screening and verifying candidate key feature points. It compares key feature points detected from different observation perspectives and retains those that can be detected from multiple perspectives to ensure their stability and reliability. The key feature point location information set is a collection of location information for stable key feature points retained after multi-view repeatability verification. In the scenario of automatic screen protector placement using a mobile phone camera, multi-view repeatability verification is performed on candidate key feature points. Since point cloud data is collected from multiple observation locations, each candidate key feature point may have different detection results under different observation perspectives. By comparing the detection results under different perspectives, the frequency of occurrence of each candidate key feature point is statistically analyzed. Points with high occurrence frequencies are considered stable key feature points. The location information of all stable key feature points is recorded to form the key feature point location information set.

[0095] As one implementation method, step S330 can be specifically implemented as the following steps S331~S334:

[0096] Step S331: Count the frequency of occurrence of each candidate key feature point under different observation perspectives and generate feature point detection frequency parameters.

[0097] The feature point detection frequency parameter describes how frequently candidate key feature points are detected under different observation views. It is obtained by statistically analyzing the occurrence frequency of each candidate key feature point across multiple observation views. This parameter helps determine the stability of candidate key feature points; the higher the occurrence frequency, the more stable the point.

[0098] Step S332: Calculate the similarity between feature descriptors of each candidate key feature point under each observation view. The similarity is calculated by the mean cosine similarity of the descriptor vectors.

[0099] A feature descriptor is a vector used to describe the features of a feature point; it contains the local geometric information of the feature point. By calculating the similarity between feature descriptors from different viewing perspectives, the stability of the feature point under different perspectives can be determined.

[0100] In a scenario where a mobile phone camera automatically locates and applies a screen protector, the similarity between the feature descriptors of each candidate key feature point under different viewing perspectives is calculated. First, a feature descriptor is calculated for each candidate key feature point under each viewing perspective, for example, using the FPFH (Fast Point Feature Histogram) descriptor. Then, for each candidate key feature point, the cosine similarity of its feature descriptor vectors under different viewing perspectives is calculated. Finally, the mean cosine similarity of all different viewing perspectives is calculated to obtain the mean similarity of the candidate key feature point.

[0101] Step S333: Construct a feature point stability evaluation index. The evaluation index is a weighted product of the detection frequency parameter and the mean similarity of the descriptor. The weight parameter is dynamically adjusted according to the feature saliency of the region where the feature point is located.

[0102] The feature point stability evaluation index is a comprehensive metric that considers both the detection frequency parameter and the mean descriptor similarity, used to assess the stability of candidate key feature points. The weighted product multiplies the detection frequency parameter and the mean descriptor similarity by weights, with the weights dynamically adjusted based on the feature saliency of the region where the feature point is located, to more accurately evaluate the stability of feature points in different regions.

[0103] In a scenario where a screen protector is automatically positioned using a mobile phone camera, a feature point stability evaluation index is constructed. First, weight parameters are determined based on the saliency of the regions where feature points are located. Regions with high saliency, such as the camera's marking area, are assigned a higher weight to the detection frequency parameter; regions with low saliency, such as the lens surface, are assigned a higher weight to the mean descriptor similarity value. Then, the detection frequency parameter and the mean descriptor similarity value are weighted and multiplied together to obtain the feature point stability evaluation index.

[0104] Step S334: Select candidate key feature points whose stability evaluation index values ​​are higher than the preset threshold as stable key feature points. Stable key feature points have high detection repeatability and feature description consistency in multiple perspectives.

[0105] The preset threshold is a pre-defined standard used to filter out candidate key feature points with high stability. Stable key feature points are candidate key feature points whose stability evaluation index values ​​are higher than the preset threshold. They have high detection repeatability and feature description consistency from multiple perspectives and can accurately represent the key structure of the workpiece.

[0106] Step S340: Perform boundary detection processing on the two-dimensional image data in the fused data group to extract the boundary contour lines between the workpiece and the background in the image.

[0107] Boundary detection is the process of finding the boundary between a workpiece and the background in a two-dimensional image. The boundary contour line refers to the outer shape boundary of the workpiece in the image, containing the shape information of the workpiece. In the scenario of automatic positioning and film application by a mobile phone camera, boundary detection processing is performed on the two-dimensional image data in the fused data set. Algorithms such as Canny or Sobel can be used. Taking the Canny edge detection algorithm as an example, firstly, the two-dimensional image is Gaussian smoothed to reduce the influence of noise. Then, the gradient magnitude and direction of the image are calculated. Next, non-maximum suppression is performed, retaining only points with local maximum gradient magnitudes. Finally, double thresholding is performed, using two thresholds to determine true edge points and potential edge points, and connecting them to form a continuous boundary contour line.

[0108] As one implementation method, step S340 can be specifically implemented as the following steps S341~S345:

[0109] Step S341: Spatial mapping processing is performed on the two-dimensional image data and the corresponding point cloud data to generate a fused image with depth information. Each pixel in the fused image is associated with its corresponding three-dimensional depth value.

[0110] Spatial mapping is the process of spatially associating two-dimensional image data with corresponding point cloud data. This process adds depth information from the point cloud data to the two-dimensional image, generating a fused image with depth information. Each pixel in the fused image is associated with a three-dimensional depth value, providing richer information for subsequent boundary detection and analysis. In the scenario of automatic screen protector placement using a mobile phone camera, spatial mapping is performed on the two-dimensional image data and its corresponding point cloud data. Using the previously established mapping relationship between point cloud coordinates and image pixel coordinates, the depth information from the point cloud data is mapped to the pixels in the two-dimensional image. For example, for a pixel (u, v) in the two-dimensional image, its corresponding three-dimensional coordinates (X, Y, Z) in the point cloud data are found through the mapping relationship, and the Z value is used as the depth value of that pixel. The depth values ​​of all pixels are added to the two-dimensional image to generate a fused image with depth information. For images from a mobile phone camera, this processing ensures that each pixel has corresponding depth information, which helps to more accurately identify boundaries.

[0111] Step S342: Perform multimodal feature enhancement processing on the fused image, and simultaneously calculate the brightness gradient of the grayscale image and the depth gradient of the depth image to generate a joint gradient response map. The boundary region between the workpiece and the background in the joint gradient response map shows significant gradient changes.

[0112] Multimodal feature enhancement processing comprehensively considers the brightness information of a grayscale image and the depth information of a depth image, enhancing image features by calculating their gradients. The brightness gradient reflects the changes in brightness in the grayscale image, while the depth gradient reflects the changes in depth in the depth image. The joint gradient response map is an image that fuses the brightness and depth gradient information. In the boundary regions between the workpiece and the background, both brightness and depth change significantly; therefore, these regions will exhibit significant gradient changes in the joint gradient response map.

[0113] In a scenario where a screen protector is automatically positioned for application by a mobile phone camera, multimodal feature enhancement processing is performed on the fused image. First, the fused image is separated into a grayscale image and a depth image. Then, the brightness gradient of the grayscale image and the depth gradient of the depth image are calculated separately. The Sobel operator can be used to calculate the gradients. For the grayscale image, its gradient components Gx and Gy in the x and y directions are calculated, and then the gradient magnitude is calculated. For depth images, their gradient components Dx and Dy in the x and y directions are also calculated, and then the depth gradient magnitude is calculated. Finally, the brightness gradient magnitude and depth gradient magnitude are weighted and summed to generate a joint gradient response map.

[0114] Step S343: Extract candidate boundary regions based on the joint gradient response map, and identify the set of pixels whose gradient response values ​​exceed the local mean as candidate boundary points through adaptive threshold segmentation.

[0115] Boundary candidate region extraction is the process of finding potential boundary regions in the joint gradient response map. Adaptive threshold segmentation is a segmentation method that automatically adjusts the threshold based on local image features, enabling more accurate identification of boundary candidate points. Boundary candidate points are pixels in the joint gradient response map whose gradient response values ​​exceed the local mean; these points may represent the boundary between the workpiece and the background.

[0116] In the scenario of automatic screen protector placement for mobile phone cameras, boundary candidate regions are extracted based on the joint gradient response map. First, the joint gradient response map is divided into multiple local regions, for example, dividing the image into 8×8 blocks. Then, the mean gradient response value of each local region is calculated. For each pixel, its gradient response value is compared with the mean of its local region. If the gradient response value of a pixel exceeds the local mean, it is considered a boundary candidate point. This adaptive thresholding segmentation operation can more accurately identify candidate points for the boundaries of the mobile phone camera, avoiding misjudgments that may occur when using a fixed threshold.

[0117] Step S344: Perform depth continuity verification on the boundary candidate points, retain the pixels whose depth value changes within the neighborhood are less than a preset range, and form boundary candidate regions with consistent depth.

[0118] Depth continuity verification is a further screening process for boundary candidate points. It determines whether a point is a true boundary point by examining the depth value changes of pixels in its neighborhood. A depth-consistent boundary candidate region refers to a region composed of pixels retained after depth continuity verification. These regions have small depth value changes and are more likely to be workpiece boundaries. In the scenario of automatic positioning and film application for mobile phone cameras, depth continuity verification is performed on boundary candidate points. For each boundary candidate point, the depth values ​​of its neighboring pixels are checked. A preset range can be set, for example, a depth value change not exceeding 0.1 mm. If the depth value change of pixels in the neighborhood of a boundary candidate point is less than 0.1 mm, the point is considered to have passed depth continuity verification and is retained. Pixels with large depth value changes in their neighborhood are removed from the boundary candidate points. This process forms depth-consistent boundary candidate regions, improving the accuracy of boundary extraction.

[0119] Step S345: Perform closure check processing on the boundary candidate regions with consistent depth, connect the continuous boundary candidate points through the contour tracking algorithm to generate a closed workpiece boundary contour line. The contour line simultaneously satisfies the dual features of image grayscale boundary and depth discontinuity boundary.

[0120] Closure checking is the process of ensuring that candidate boundary regions form closed contours, while contour tracking algorithms are used to connect continuous candidate boundary points to form complete contour lines. A closed workpiece boundary contour line refers to a continuous, closed boundary line obtained after closure checking and contour tracking. It simultaneously satisfies the dual characteristics of image grayscale boundary and depth discontinuity boundary, and can more accurately represent the shape of the workpiece.

[0121] In the scenario of automatic positioning and screen protector application for mobile phone cameras, closure checks are performed on candidate boundary regions with consistent depth. Zhang-Suen or Guo-Hall thinning algorithms can be used to refine these candidate boundary regions, making the boundaries clearer. Then, a contour tracking algorithm, such as a boundary tracking algorithm, is used to connect adjacent candidate boundary points sequentially according to certain rules, starting from a single candidate boundary point, until a closed contour line is formed. During the connection process, it is essential to ensure that the contour line simultaneously satisfies the dual characteristics of image grayscale boundaries and depth discontinuity boundaries. For the boundaries of a mobile phone camera, this processing yields an accurate closed boundary contour line that accurately depicts the shape of the camera.

[0122] Step S350: Transform the image boundary contour lines to the three-dimensional spatial coordinate system of the global point cloud data through projection mapping relationship, and generate a three-dimensional contour line aligned with the point cloud data.

[0123] The projection mapping relationship is the previously established mapping relationship between point cloud coordinates and image pixel coordinates. This relationship allows the boundary contour lines in a 2D image to be transformed into the global point cloud data coordinate system in 3D space, generating a 3D contour line aligned with the point cloud data. This 3D contour line can more accurately describe the position and shape of a workpiece in 3D space. In the scenario of automatic positioning and film application for a mobile phone camera, the image boundary contour lines are transformed into the 3D coordinate system of the global point cloud data using the projection mapping relationship. For each pixel on the image boundary contour line, its corresponding 3D coordinates in the global point cloud data are found according to the projection mapping relationship. Connecting the corresponding 3D coordinates of all pixels generates a 3D contour line aligned with the point cloud data. For example, for a pixel (u, v) on the boundary contour line of a mobile phone camera image, its corresponding 3D coordinates (X, Y, Z) are found through the projection mapping relationship, and these coordinates are added to the point set of the 3D contour line. Finally, a 3D contour line that accurately describes the shape of the mobile phone camera is obtained, and this contour line is spatially aligned with the global point cloud data.

[0124] Step S360: Perform continuity analysis on the three-dimensional contour lines, extract continuous contour line segments with consistent features, and form a set of continuous contour line feature descriptions.

[0125] Continuity analysis is a process of checking and analyzing 3D contour lines to determine their continuity and feature consistency. Continuous and feature-consistent contour segments refer to continuous line segments with the same geometric features within a 3D contour line; these segments represent the shape characteristics of different parts of a workpiece. A continuous contour feature description set is a collection of feature descriptions for these continuous and feature-consistent contour segments, providing more detailed information for subsequent feature matching and positioning. In the scenario of automatic positioning and film application for mobile phone cameras, continuity analysis is performed on the 3D contour lines. First, the 3D contour line is divided into multiple small segments, and then the continuity and feature consistency of each segment are checked. Continuity and feature consistency can be determined by calculating the distance and angle changes between adjacent points. If the distance between adjacent points is less than a preset threshold, and the angle change is also less than a preset threshold, then the segment is considered continuous and feature-consistent. All continuous and feature-consistent contour segments are extracted to form a continuous contour feature description set. The three-dimensional contour line of a mobile phone camera can be divided into different continuous contour line segments with consistent characteristics, such as the lens edge and the surrounding structure. The length, curvature and other features of each line segment can be described to form a set of continuous contour line feature descriptions.

[0126] Step S370: Merge the set of key feature point location information with the set of continuous contour line feature descriptions to form a set of workpiece feature identifiers.

[0127] The purpose of merging the set of key feature point location information with the set of continuous contour line feature descriptions is to integrate the information of key feature points and continuous contour lines on the workpiece surface to form a more comprehensive and accurate set of workpiece feature identifiers.

[0128] In the scenario of automatic positioning and film application using a mobile phone camera, the set of key feature point location information obtained in step S330 is merged with the set of continuous contour line feature descriptions obtained in step S360. The location information of the key feature points and the feature description information of the continuous contour lines are stored in the same data structure to form a workpiece feature identifier set. For example, the three-dimensional coordinates of the key feature points and the feature description information such as the length and curvature of the continuous contour lines are stored in a list or dictionary to facilitate subsequent feature matching and positioning operations.

[0129] Step S400: Perform feature matching association processing between the workpiece feature identifier set and the pre-stored standard workpiece feature template to generate a feature matching correspondence set. The feature matching correspondence set includes the positional correspondence of key feature points and the shape matching relationship of contour lines.

[0130] Feature matching and association processing is the process of comparing and matching a set of workpiece feature identifiers with pre-stored standard workpiece feature templates. This process allows the identification of correspondences between workpiece features and standard template features. The feature matching correspondence set records the positional correspondences of key feature points and the shape matching relationships of contour lines.

[0131] As one implementation method, step S400 can be specifically implemented as the following steps S410~S460:

[0132] Step S410: Extract the set of standard key feature points and the set of standard contour lines from the pre-stored standard workpiece feature template.

[0133] The pre-stored standard workpiece feature template is stored in the system beforehand and represents the feature information of the workpiece under ideal conditions. The standard key feature point set is a set of representative points in the template that can reflect the important structure of the workpiece. The positions and features of these points are relatively stable in different production batches and application scenarios. The standard contour line set is a set of continuous curves in the template that describe the outer boundary of the workpiece.

[0134] Step S420: Calculate the spatial distance between the key feature points in the workpiece feature identifier set and each point in the standard key feature point set, and establish an initial matching pair.

[0135] Spatial distance is the distance between two points in three-dimensional space. By calculating the spatial distance between key feature points in the workpiece feature identifier set and each point in the standard key feature point set, a preliminary correspondence between them can be determined. Initial matching pairs are pairs composed of key feature points that are relatively close to each other; these pairs will serve as the basis for further verification and screening.

[0136] In the scenario of automatic positioning and film application using a mobile phone camera, for each key feature point in the workpiece feature identifier set, the spatial distance between it and each point in the standard key feature point set is calculated. The Euclidean distance formula can be used to calculate the spatial distance. For a given key feature point of the mobile phone camera, the distance between it and all key feature points in the standard template is calculated, and the point with the smallest distance is paired with that key feature point to form an initial matching pair.

[0137] Step S430: Perform spatial relationship consistency verification on the initial matching pairs, and select matching pairs that satisfy the relative positional relationship as valid key feature point matching pairs to form the positional correspondence of key feature points.

[0138] Spatial relationship consistency verification ensures that the relative spatial positions of key feature points in the initial matching pair match with their corresponding points in the standard template. Valid key feature point matching pairs are those that have been verified and meet the relative positional relationship requirements; they more accurately reflect the correspondence between the workpiece and the standard template. The positional correspondence of key feature points is a set composed of these valid key feature point matching pairs.

[0139] In scenarios where mobile phone cameras automatically locate screen protector placement, spatial relationship consistency verification is performed on the initial matching pairs. This can be achieved by comparing the relative positional relationships of neighboring feature points of key feature points in the initial matching pairs.

[0140] As one implementation method, step S430 can be specifically implemented as the following steps S431~S434:

[0141] Step S431: Extract the set of neighborhood feature points of the key feature points in the initial matching pairs of the benchmark data group, and construct a local topology structure containing the relative positions of the key feature points and their neighbors.

[0142] A neighborhood feature point set refers to the set of other feature points within a certain range around a key feature point. The relative positional relationships of these points can reflect the local features of the key feature point. Local topology is a structure formed by the relative positional relationships between the key feature point and its neighborhood feature points; it can be represented by graphs or matrices.

[0143] In the scenario of automatic screen protector placement using a mobile phone camera, for key feature points in the initial matching pair of the baseline data set, a set of neighboring feature points is extracted. The neighborhood range can be determined by setting a neighborhood radius; for example, all feature points within a 1mm radius centered on the key feature point are considered as neighboring feature points. If there is a key feature point P in the initial matching pair of the baseline data set, its coordinates are (x... p ,y p ,z p The process iterates through all feature points in the workpiece feature identifier set, calculating the distance between each point and P. If the distance is less than 1 mm, the point is added to the neighborhood feature point set. Then, a local topology is constructed containing the relative positions of the key feature point P and its neighboring points. This can be represented by a matrix, where each row represents a neighboring feature point, and the columns represent the relative distances between that neighboring feature point and the key feature point P in the X, Y, and Z directions, respectively. For example, for a neighboring feature point Q, its coordinates are (x...). q ,y q ,z q If ), then the corresponding row element in the matrix is ​​(x q -x p ,y q -y p ,z q -z p ).

[0144] Step S432: Extract the set of neighborhood feature points of the corresponding key feature points in the initial matching pairs of other observation locations, and construct a local topology of the same size.

[0145] For key feature points in the initial matching pairs of other observation locations, it is also necessary to extract their neighborhood feature point sets and construct a local topology, ensuring that the scale of the local topology is the same as that of the key feature points in the initial matching pairs of the baseline data set, for subsequent comparison and verification. In the scenario of automatic positioning and screen protector application by mobile phone cameras, for the corresponding key feature points in the initial matching pairs of other observation locations, the same method as in step S431 is used to extract the neighborhood feature point sets. For example, for a key feature point R in an initial matching pair of other observation locations, a neighborhood radius of 1 mm is used to traverse the feature points at that observation location, and points less than 1 mm away from R are identified as the neighborhood feature point set. Then, a local topology of the same scale is constructed. If the neighborhood feature point set of the key feature point in the baseline data set has n points, then when constructing the local topology of the corresponding key feature point at other observation locations, it is also necessary to ensure that the neighborhood feature point set has n points as much as possible. If there are fewer than n points in the neighborhood, interpolation or approximation methods can be used to supplement them; if there are too many points, they can be filtered to select the n points closest to the key feature point. Similarly, matrices are used to represent the local topology for subsequent comparisons.

[0146] Step S433: Calculate the direction consistency and distance deviation values ​​of the corresponding relative position vectors in the two sets of local topologies. The direction consistency is measured by the cosine of the angle between the vectors, and the distance deviation value is measured by the absolute value of the difference in vector lengths.

[0147] Directional consistency refers to the degree of similarity in the directions of corresponding relative position vectors in two sets of local topologies. It can be intuitively reflected by the cosine of the angle between the vectors. Distance deviation refers to the difference in length between corresponding relative position vectors in two sets of local topologies. It can be reflected by the absolute value of the difference in vector lengths to indicate whether the lengths of the two vectors are similar.

[0148] Step S434: Set the orientation consistency threshold and the distance deviation threshold, and retain the initial matching pairs that simultaneously satisfy the orientation consistency higher than the threshold and the distance deviation lower than the threshold as valid key feature point matching pairs. Valid matching pairs simultaneously satisfy the requirements of global position correspondence and local structure consistency.

[0149] The orientation consistency threshold and distance deviation threshold are pre-set standards used to filter out initial matching pairs that meet the spatial relationship consistency requirements. Valid key feature point matching pairs are initial matching pairs that simultaneously meet the requirements of orientation consistency being higher than the threshold and distance deviation being lower than the threshold. They correspond globally and are consistent locally, accurately reflecting the correspondence between key feature points of the workpiece and the standard template.

[0150] Step S440: Perform shape similarity measurement on the continuous contour lines in the workpiece feature identifier set and the contour lines in the standard contour line set, and calculate the feature difference value between the two at equidistant sampling points.

[0151] Shape similarity measurement is a method used to evaluate the similarity in shape between continuous contour lines in a set of workpiece feature identifiers and contour lines in a set of standard contour lines. Equidistant sampling points are points selected at certain intervals along the contour lines. By comparing the features at these equidistant sampling points, the feature difference value between the two can be calculated. The smaller the feature difference value, the more similar the shapes of the two contour lines are. In the scenario of automatic positioning and film application for mobile phone cameras, shape similarity measurement is performed on continuous contour lines in the set of workpiece feature identifiers and contour lines in the set of standard contour lines. First, equidistant sampling is performed on the contour lines. Then, the feature difference value at the two sets of equidistant sampling points is calculated, which can be done using Euclidean distance. The feature difference value is obtained by summing the Euclidean distances of all pairs of equidistant sampling points.

[0152] Step S450: Select the contour line pair with the smallest feature difference value as the effective contour line matching pair to form the shape matching relationship of the contour lines.

[0153] An effective contour line matching pair refers to a contour line pair with the smallest feature difference value. They are most similar in shape and can accurately reflect the contour line correspondence between the workpiece and the standard template. The shape matching relationship of contour lines is a set composed of these effective contour line matching pairs.

[0154] In the scenario of automatic positioning and film application for mobile phone cameras, the shape similarity of all continuous contour lines in the workpiece feature identifier set and all contour lines in the standard contour line set is measured to obtain a series of feature difference values. These feature difference values ​​are compared, and the contour line pair corresponding to the smallest feature difference value is selected as the valid contour line matching pair. All valid contour line matching pairs are organized into a set to form the shape matching relationship of the contour lines.

[0155] Step S460: Merge the valid key feature point matching pairs and the valid contour line matching pairs to form a feature matching correspondence set.

[0156] Merging valid key feature point matching pairs and valid contour line matching pairs aims to integrate the correspondences between key feature points and contour lines between the workpiece and the standard template, forming a more comprehensive and accurate set of feature matching correspondences. This set contains the correspondence information between the workpiece and the standard template in terms of key features and external contours.

[0157] For example, the set of valid key feature point matching pairs obtained in step S434 and the set of valid contour line matching pairs obtained in step S450 can be merged. They can be stored in a unified data structure, for example, using a list, where the elements of the list are valid key feature point matching pairs and valid contour line matching pairs, respectively.

[0158] Step S500: Calculate the three-dimensional positioning coordinate information of the target workpiece in the scene based on the feature matching correspondence set. The three-dimensional positioning coordinate information includes the workpiece centroid coordinates and spatial attitude angle parameters.

[0159] The calculation of the target workpiece's 3D positioning coordinates in the scene based on the feature matching correspondence set determines the specific position and orientation of the target workpiece in the current scene by utilizing the feature correspondence between the workpiece and the standard template. The workpiece's centroid coordinates refer to the position of its center of gravity in 3D space, reflecting its overall position information. Spatial orientation angle parameters describe the workpiece's rotational state in 3D space, including rotation angles around the X, Y, and Z axes, accurately representing the workpiece's orientation.

[0160] As one implementation method, step S500 can be specifically implemented as the following steps S510~S570:

[0161] Step S510: Select multiple sets of valid key feature point matching pairs from the positional correspondence of key feature points.

[0162] Selecting multiple sets of valid key feature point matching pairs from the positional correspondence of key feature points is to provide sufficient information to calculate the positional transformation parameters of the target workpiece. Multiple sets of valid key feature point matching pairs can more accurately reflect the positional correspondence between the workpiece and the standard template, thereby improving the accuracy of the calculation results.

[0163] In the scenario of automatic positioning and screen protector application for mobile phone cameras, multiple sets of valid key feature point matching pairs are selected from the positional correspondence of key feature points obtained in step S434. For example, matching pairs between the lens center point and the lens center point in the standard template, and matching pairs between the lens periphery marker vertices and their corresponding vertices in the standard template are selected. If there are 10 sets of valid key feature point matching pairs in the positional correspondence of key feature points, 5 representative matching pairs are selected, such as matching pairs located at different positions on the lens, to ensure that the positional relationship between the mobile phone camera and the standard template can be fully reflected.

[0164] Step S520: Using a coordinate transformation algorithm, the scene coordinates and standard coordinates of the effective key feature point matching pairs are used to calculate the position transformation parameters of the target workpiece relative to the standard workpiece.

[0165] Coordinate transformation algorithms are used to calculate the positional transformation of a target workpiece relative to a standard workpiece. By matching the scene coordinates and standard coordinates of effective key feature points, the transformation parameters such as translation, rotation, and scaling of the target workpiece in space can be determined. The positional transformation parameters can be represented by a transformation matrix that converts the coordinates of the standard workpiece to the coordinates of the target workpiece.

[0166] In the scenario of automatic positioning and screen protector application using a mobile phone camera, coordinate transformation algorithms, such as ICP (Iterative Closest Point) or SVD (Singular Value Decomposition), are used to calculate the position transformation parameters of the target workpiece relative to the standard workpiece using the scene coordinates and standard coordinates of selected effective key feature point matching pairs. Taking the SVD algorithm as an example, for the selected 5 sets of effective key feature point matching pairs, their scene coordinates and standard coordinates are used to form two point sets, P and Q. First, the centroid C of point sets P and Q is calculated. p and C q Then, the point sets P and Q are centroid-free to obtain new point sets P' and Q'. Next, the covariance matrices of P' and Q' are calculated. , where p i 'and q i ' are points in P' and Q' respectively. Singular value decomposition is performed on the covariance matrix H to obtain... Finally, calculate the rotation matrix R = VU. TTranslation vector t=C q -RC p The rotation matrix R and the translation vector t constitute the position transformation parameters of the target workpiece relative to the standard workpiece.

[0167] Step S530: Based on the position transformation parameters, transform the centroid coordinates of the standard workpiece to the scene coordinate system to obtain the centroid coordinates of the target workpiece.

[0168] Transforming the centroid coordinates of a standard workpiece to the scene coordinate system based on position transformation parameters involves applying these parameters to the centroid coordinates of the standard workpiece, thus achieving the conversion from the standard coordinate system to the scene coordinate system. The centroid coordinates of the target workpiece are its center of gravity position in the scene coordinate system, accurately reflecting its overall position within the current scene.

[0169] In a scenario where the mobile phone camera automatically positions the screen protector, the centroid coordinates C of a standard mobile phone camera are known. s The position transformation parameters (rotation matrix R and translation vector t) calculated in step S520 are used. The centroid coordinates C of the standard workpiece are then... s Perform the transformation according to the transformation formula C. t =RC s +t, where Ct is the centroid coordinate of the target workpiece in the scene coordinate system. For example, the centroid coordinates of a standard mobile phone camera are C... s =(x s ,y s ,z s Rotation matrix Translation vector t=(t x ,t y ,t z If the centroid coordinates of the target workpiece are given, then... .

[0170] Step S540: Decompose the position transformation parameters using the attitude calculation algorithm and extract the rotation angle of the target workpiece around each coordinate axis as the spatial attitude angle parameter.

[0171] Attitude calculation algorithms decompose rotational information from position transformation parameters to obtain the rotation angles of the target workpiece around each coordinate axis. Spatial attitude angle parameters are important parameters describing the rotational state of the target workpiece in three-dimensional space, and they can accurately represent the workpiece's attitude.

[0172] As one implementation method, step S540 can be specifically implemented as the following steps S541~S547:

[0173] Step S541: Separate the rotation component and translation component from the position transformation parameters. The rotation component is used to represent the spatial orientation transformation relationship of the workpiece.

[0174] Position transformation parameters typically include rotation and translation components. The rotation component describes the workpiece's rotational state in three-dimensional space and can be represented by a rotation matrix. The translation component describes the workpiece's translational state in three-dimensional space and can be represented by a translation vector. Separating the rotation and translation components from the position transformation parameters allows for separate processing of the rotation component to extract the workpiece's spatial orientation angle parameters. For the position transformation parameters (rotation matrix R and translation vector t) calculated in step S520, the rotation matrix R is directly used as the rotation component, and the translation vector t as the translation component. The rotation matrix R can represent the rotation state of the mobile phone camera relative to the standard template. By further processing the rotation matrix R, the rotation angles of the mobile phone camera around each coordinate axis can be obtained.

[0175] Step S542: Convert the shape matching relationship of the contour lines into attitude constraints. The constraints include the relative position consistency and orientation alignment requirements of the matching contour lines in three-dimensional space.

[0176] Converting the shape matching relationship of contour lines into attitude constraints allows for the use of contour line shape information to further constrain the workpiece's attitude. The requirement for relative position consistency of matching contour lines in three-dimensional space means that the relative positions of continuous contour lines in the workpiece feature set and matching contour lines in the standard contour line set should remain consistent in three-dimensional space. The requirement for direction alignment means that the tangent directions of two matching contour lines should be as consistent as possible.

[0177] In the scenario of automatic positioning and screen protector application for mobile phone cameras, the valid contour matching pairs obtained in step S450 are converted into attitude constraints. For example, for a circular contour matching pair of a mobile phone camera, it is required that the centers of the two circular contours are relatively consistent in three-dimensional space, and the tangent directions are as similar as possible at corresponding points. The attitude constraints can be determined by calculating the relative positions and tangent directions of equidistant sampling points on the matching contours. For each pair of equidistant sampling points on the matching contours, the relative position deviation and the angle between their tangent directions are calculated. If the relative position deviation exceeds a certain threshold or the angle between their tangent directions exceeds a certain range, the attitude constraints are considered not met.

[0178] Step S543: Construct a multi-feature joint optimization model, with the positional error of key feature point matching and the shape error of contour line matching as the dual objective optimization functions.

[0179] The multi-feature joint optimization model comprehensively considers the positional error of key feature point matching and the shape error of contour line matching to improve the accuracy of workpiece posture calculation. The bi-objective optimization function is the core of this model; it treats the positional error of key feature point matching and the shape error of contour line matching as two objectives. By optimizing the values ​​of these two objective functions, a more accurate workpiece posture can be obtained.

[0180] In the scenario of automatic screen protector placement using a mobile phone camera, a multi-feature joint optimization model is constructed. The positional error of key feature point matching can be measured by calculating the Euclidean distance between the effective key feature point matching pairs in the scene coordinates and the transformed standard coordinates. The shape error of contour line matching can be represented by the feature difference value calculated in step S440. The bi-objective optimization function can be expressed as follows: ,in It is the positional error of key feature point matching. It represents the shape error of the contour matching, while w1 and w2 are weighting coefficients that are adjusted according to the importance of key feature points and contours in the localization process.

[0181] Step S544: Adjust the parameters of the rotation component through an iterative optimization algorithm to minimize the total error of the multi-feature joint optimization function, and obtain the optimized rotation component.

[0182] Iterative optimization algorithms are used to continuously adjust the parameters of the rotational components to find a method that minimizes the total error of the multi-feature joint optimization function. Through multiple iterations, the parameters of the rotational components are gradually optimized so that the positional error of key feature point matching and the shape error of contour line matching are minimized, thereby obtaining a more accurate workpiece posture.

[0183] In the scenario of automatic screen protector placement using a mobile phone camera, iterative optimization algorithms, such as gradient descent or the Levenberg-Marquardt algorithm, are used to adjust the parameters of the rotation components separated in step S541. Taking gradient descent as an example, the parameters of the rotation components are first initialized, and then the gradient of the multi-feature joint optimization function is calculated under the current parameters. Based on the direction of the gradient, the parameters of the rotation components are updated with a certain step size. This process is repeated until the total error of the multi-feature joint optimization function converges to a small value.

[0184] Step S545: Perform orthogonality constraint processing on the optimized rotation components to ensure that the rotation components satisfy the mathematical properties of orthogonal matrices.

[0185] Orthogonality constraint processing involves further adjusting the optimized rotation components to ensure they satisfy the mathematical properties of orthogonal matrices. An orthogonal matrix is ​​a matrix whose transpose is equal to its inverse. In 3D rotations, the rotation matrix must be orthogonal to guarantee the correctness of the rotation operation.

[0186] In the scenario of automatic screen protector placement using a mobile phone camera, the optimized rotation component obtained in step S544 is subjected to orthogonality constraints. The QR decomposition method can be used to decompose the rotation component into the product of an orthogonal matrix Q and an upper triangular matrix R, and then the orthogonal matrix Q is used as the rotation component that satisfies the orthogonality constraints. For example, for the optimized rotation component R... opt QR decomposition yields R opt =QR, taking the orthogonal matrix Q as the new rotation component, ensures that the rotation component satisfies the mathematical properties of the orthogonal matrix, thus guaranteeing the accuracy of the mobile phone camera pose calculation.

[0187] Step S546: Decompose the orthogonalized rotation component into basic rotation operations around the X-axis, Y-axis, and Z-axis, and calculate the rotation angle corresponding to each basic rotation operation as the spatial attitude angle parameter.

[0188] Decomposing the orthogonalized rotation components into basic rotation operations around the X, Y, and Z axes is to break down the 3D rotation into three fundamental rotation operations, allowing for the calculation of rotation angles around each coordinate axis. The rotation angles corresponding to each basic rotation operation are the spatial attitude angle parameters, which accurately describe the workpiece's attitude in 3D space.

[0189] In the scenario of automatic screen protector placement using a mobile phone camera, the orthogonalized rotation component Q obtained in step S545 is decomposed into basic rotation operations around the X, Y, and Z axes. Euler angle decomposition can be used to represent the rotation matrix Q as a rotation around the X-axis. Rotation around the Y-axis Rotation around the Z-axis The product of the three fundamental rotation matrices, i.e. By solving this equation, the rotation angles around the X, Y, and Z axes can be obtained. .

[0190] Step S547: Perform multi-view consistency verification on the extracted spatial attitude angle parameters, compare the differences in attitude angles calculated under different observation positions, and retain the angle parameters with differences less than a preset threshold as the final result.

[0191] Multi-view consistency verification is performed to ensure the consistency of extracted spatial attitude angle parameters across different observation locations. The differences in attitude angles calculated at different observation locations are compared. If the difference is less than a preset threshold, the angle parameter is considered reliable and retained as the final result. The preset threshold is a pre-defined standard used to determine whether the attitude angle difference is within an acceptable range.

[0192] In the scenario of automatic screen protector placement using a mobile phone camera, the spatial attitude angle parameters extracted in step S546 undergo multi-view consistency verification. Since the attitude angle of the mobile phone camera can be calculated from different observation positions, the differences in the calculated attitude angles from these different observation positions are compared. For example, the rotation angle around the X-axis calculated from the observation position directly above the lens is... The rotation angle around the X-axis calculated from the observation position diagonally above the lens is: Calculate the difference value A preset threshold of 2° was set. Since the difference was less than the preset threshold, one of the two angle parameters was retained as the final result. The same verification process was performed on the rotation angles around the Y-axis and Z-axis to obtain reliable spatial attitude angle parameters.

[0193] Step S550: Verify the error of the shape matching relationship of the contour lines and calculate the positional deviation of the matching contour lines in the scene coordinate system.

[0194] Error verification of the shape matching relationship of the contour lines is to evaluate the accuracy of the matched contour lines in the scene coordinate system. Position deviation refers to the difference between the actual position and the ideal position of the matched contour lines in the scene coordinate system. By calculating the position deviation, it can be determined whether the currently calculated workpiece posture and position are accurate.

[0195] In the scenario of automatic positioning and screen protector application for the mobile phone camera, the positional deviation of the effective contour matching pairs obtained in step S450 is calculated in the scene coordinate system. For each pair of equidistant sampling points on the matching contour line, the difference between their actual distance and ideal distance in the scene coordinate system is calculated. For example, for the circular contour matching pair of the mobile phone camera, the distance deviation of the corresponding equidistant sampling points on the matching contour line is calculated in the scene coordinate system. The distance deviations of all equidistant sampling points are summed and averaged to obtain the positional deviation of the matching contour line in the scene coordinate system.

[0196] Step S560: If the position deviation meets the preset requirements, the currently calculated centroid coordinates and spatial attitude angle parameters are retained as the final three-dimensional positioning coordinate information.

[0197] If the positional deviation meets the preset requirements, it indicates that the calculated workpiece posture and position are accurate. Therefore, the calculated centroid coordinates and spatial posture angle parameters are retained as the final three-dimensional positioning coordinate information. The preset requirements are a pre-defined standard used to determine whether the positional deviation is within an acceptable range.

[0198] Step S570: If the position deviation does not meet the preset requirements, then reselect a valid key feature point matching pair and repeat the coordinate transformation calculation process until the error requirements are met.

[0199] If the positional deviation does not meet the preset requirements, it indicates that the currently calculated workpiece posture and position are inaccurate and need to be recalculated. A new pair of valid key feature points is selected, and the coordinate transformation calculation process is repeated. Through continuous adjustment and optimization, the positional deviation of the matched contour line in the scene coordinate system meets the preset requirements.

[0200] In the scenario of automatic positioning and screen protector application for a mobile phone camera, if the positional deviation of the matching contour line calculated in step S550 in the scene coordinate system is greater than 0.2 mm, it does not meet the preset requirements. Then, a new set of valid key feature point matching pairs is selected from the positional correspondence of the key feature points obtained in step S434. For example, if 5 sets of valid key feature point matching pairs were previously selected, another 5 sets are now selected. Then, the coordinate transformation calculation process of steps S520-S560 is repeated, including calculating position transformation parameters, decomposing rotational components, calculating spatial attitude angle parameters, and verifying positional deviations. This process is repeated until the positional deviation of the matching contour line in the scene coordinate system meets the preset requirements, obtaining accurate final 3D positioning coordinate information. In this way, the precision workpiece (mobile phone camera) positioning based on 3D vision is completed, providing accurate position and attitude information for subsequent operations such as screen protector application.

[0201] It is understood that the various algorithms involved in the above descriptions of the embodiments of the present invention, such as Euclidean distance algorithm, cosine distance algorithm, pose calculation algorithm, SVD algorithm, etc., can all be obtained from relevant content in the prior art. To save space, they will not be elaborated on in the embodiments of the present invention. In addition, those skilled in the art can supplement the details based on common knowledge in the art when implementing the solution of the present invention. For example, they can use normalization to eliminate dimensional conflicts before feature fusion, use interpolation to eliminate dimensional differences, reasonably set the threshold based on historical data, experience or business scenario requirements, train the model based on a general model training method, set the number of layers in the model structure based on actual needs, select the activation function, etc. The present invention will not provide redundant descriptions of overly detailed implementation processes here.

[0202] Figure 2 A hardware entity diagram of a computer system provided as an embodiment of the present invention, such as... Figure 2 As shown, the hardware entity of the computer system 1000 includes a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can run on the processor 1001, and the processor 1001 executes the program to implement the steps in the method of any of the above embodiments.

Claims

1. A precision workpiece positioning method based on three-dimensional vision, characterized in that, The method includes: Acquire a set of scene 3D visual data containing the target workpiece, the set of scene 3D visual data consists of workpiece surface point cloud data acquired synchronously from multiple perspectives and 2D image data from corresponding perspectives; Multimodal data fusion processing is performed on the scene's 3D visual data set to generate a spatially aligned fused data set. This fused data set includes a mapping relationship between point cloud coordinates and 2D image pixel coordinates. Specifically, this includes: selecting any point cloud-image data pair from the scene's 3D visual data set as a reference data set; extracting feature points from the 2D images of the reference data set; performing feature matching between 2D image data from other observation locations and the 2D image data of the reference data set using a feature matching algorithm to obtain the transformation relationship between images at different observation locations; calculating coordinate transformation parameters of the point cloud data from other observation locations relative to the point cloud data of the reference data set based on the transformation relationship and the spatial reference datum of each observation location; transforming the point cloud data from other observation locations to the spatial coordinate system of the reference data set using the coordinate transformation parameters, and then stitching it with the point cloud data of the reference data set to generate global point cloud data; performing projection transformation processing on the global point cloud data and the 2D image data from each observation location, projecting the 3D coordinates of the point cloud onto the pixel coordinate system of the corresponding 2D image through camera parameters, establishing a mapping relationship between point cloud coordinates and image pixel coordinates, and forming a spatially aligned fused data set. Extract a set of workpiece feature identifiers from the fused data set. The set of workpiece feature identifiers includes the location information of key feature points on the workpiece surface and the feature description of continuous contour lines. The workpiece feature identifier set is matched and associated with a pre-stored standard workpiece feature template to generate a feature matching correspondence set, which includes the positional correspondence of key feature points and the shape matching relationship of contour lines. Based on the feature matching correspondence set, the three-dimensional positioning coordinate information of the target workpiece in the scene is calculated. The three-dimensional positioning coordinate information includes the workpiece centroid coordinates and spatial attitude angle parameters. The step of performing feature matching between two-dimensional image data from other observation locations and two-dimensional image data from the reference data set to obtain the transformation relationship between images from different observation locations includes: Feature points are extracted from the two-dimensional image data of the baseline data set, and the three-dimensional coordinates of each feature point in the corresponding point cloud data are simultaneously obtained as feature space anchor points. Perform the same feature point extraction and feature space anchor point acquisition operations on the two-dimensional image data of other observation locations to generate a feature point-space anchor point association set for the corresponding observation location; Calculate the image feature descriptor similarity between feature points in the baseline data set and feature points in other observation locations to generate an initial similarity matching matrix; Based on the three-dimensional coordinate information of the feature space anchor points, the spatial Euclidean distance between the feature points of the benchmark data group and the feature points of other observation locations is calculated, and a spatial distance matching matrix is ​​generated. The initial similarity matching matrix and the spatial distance matching matrix are weighted and fused to generate a comprehensive matching score matrix. The weight parameters of the weighted fusion are dynamically adjusted according to the texture complexity of the region where the feature points are located. Feature point pairs with a comprehensive matching score higher than a preset threshold are selected as candidate matching pairs, and the candidate matching pairs simultaneously meet the requirements of image feature similarity and spatial proximity. Perform topological consistency verification on candidate matching pairs to ensure that the relative positional relationship between feature points in the baseline data group is consistent with the relative positional relationship between feature points in other observation locations. Candidate matching pairs that pass the topological consistency verification are retained as valid matching pairs, and the transformation relationship parameters between images at different observation locations are solved based on the coordinate information of the valid matching pairs.

2. The method of claim 1, wherein, The acquisition of the scene 3D visual data set containing the target workpiece includes: The three-dimensional vision acquisition device is controlled to acquire data at multiple preset observation positions around the workpiece, and fixed device pose parameters are set for each observation position; At each observation position, point cloud data of the workpiece surface is acquired by a lidar sensor, and the point cloud data contains the three-dimensional spatial coordinate information of each point; At the same observation position, two-dimensional image data of the corresponding viewpoint are acquired synchronously by an RGB camera. The two-dimensional image data includes the texture and color information of the workpiece surface. Record the device pose parameters at each observation location as a spatial reference benchmark, which includes the device's position and orientation information relative to the scene's fixed coordinate system; The point cloud data and two-dimensional image data collected at each observation location are stored according to the corresponding spatial reference datum, forming a set of scene three-dimensional visual data composed of multiple sets of point cloud-image data pairs.

3. The precision workpiece positioning method based on three-dimensional vision as described in claim 1, characterized in that, The step of extracting the workpiece feature identifier set from the fused data set includes: Noise filtering is performed on the global point cloud data in the fused data group to eliminate discrete noise points in the point cloud data. The feature detection algorithm identifies regions with significant features in the smoothed point cloud data, and the center point of the region is used as a candidate key feature point. Candidate key feature points are verified repeatedly from multiple perspectives, and stable key feature points that can be detected from multiple observation perspectives are retained to form a set of key feature point location information. Boundary detection processing is performed on the two-dimensional image data in the fused data set to extract the boundary contour lines between the workpiece and the background in the image; The image boundary contour is transformed to the three-dimensional spatial coordinate system of the global point cloud data through projection mapping relationship, generating a three-dimensional contour that is aligned with the point cloud data; The continuity analysis of the three-dimensional contour lines is performed to extract continuous contour line segments with consistent features, forming a set of continuous contour line feature descriptions. The set of key feature point location information and the set of continuous contour line feature descriptions are merged to form the workpiece feature identifier set.

4. The precision workpiece positioning method based on three-dimensional vision as described in claim 3, characterized in that, The step of performing feature matching and association processing between the workpiece feature identifier set and the pre-stored standard workpiece feature template to generate a feature matching correspondence set includes: Extract the set of standard key feature points and the set of standard contour lines from the pre-stored standard workpiece feature template; Calculate the spatial distance between the key feature points in the workpiece feature identifier set and each point in the standard key feature point set, and establish initial matching pairs; The initial matching pairs are verified for spatial relationship consistency. Matching pairs that satisfy the relative positional relationship are selected as valid key feature point matching pairs, thus forming the positional correspondence of key feature points. The shape similarity between the continuous contour lines in the workpiece feature identifier set and the contour lines in the standard contour line set is measured, and the feature difference values ​​between the two at equidistant sampling points are calculated. The contour line pairs with the smallest feature difference values ​​are selected as valid contour line matching pairs to form shape matching relationships of contour lines. The effective key feature point matching pairs and effective contour line matching pairs are merged to form a feature matching correspondence set.

5. The method of claim 4, wherein, The calculation of the three-dimensional positioning coordinates of the target workpiece in the scene based on the feature matching correspondence set includes: Select multiple sets of valid key feature point matching pairs from the positional correspondence of key feature points; By using a coordinate transformation algorithm, the scene coordinates and standard coordinates of the effective key feature point matching pairs are used to calculate the position transformation parameters of the target workpiece relative to the standard workpiece. Based on the position transformation parameters, the centroid coordinates of the standard workpiece are transformed to the scene coordinate system to obtain the centroid coordinates of the target workpiece. The position transformation parameters are decomposed by the attitude calculation algorithm, and the rotation angle of the target workpiece around each coordinate axis is extracted as the spatial attitude angle parameter. Verify the error of the shape matching relationship of the contour lines and calculate the positional deviation of the matched contour lines in the scene coordinate system; If the position deviation meets the preset requirements, the currently calculated centroid coordinates and spatial attitude angle parameters are retained as the final three-dimensional positioning coordinate information. If the positional deviation does not meet the preset requirements, then select a new pair of valid key feature points and repeat the coordinate transformation calculation process until the error requirements are met.

6. The precision workpiece positioning method based on three-dimensional vision as described in claim 2, characterized in that, The controlled 3D vision acquisition device acquires data at multiple preset observation positions around the workpiece. Each observation position is configured with fixed device pose parameters, including: The workpiece surface is pre-scanned to generate preliminary point cloud data containing surface feature distribution, which includes texture complexity and curvature change rate information. An observation location planning model is constructed based on the surface feature distribution information. The observation location planning model takes the coverage integrity of feature-rich areas and the redundancy control of feature-sparse areas as the optimization objectives. The target observation location set around the workpiece is dynamically determined by the observation location planning model, and the observation point density in the feature-rich region of the target observation location set is higher than that in the feature-sparse region. At each target observation position, the lens orientation of the 3D vision acquisition device is adjusted by the pose calibration module so that the lens optical axis and the normal vector of the corresponding area of ​​the workpiece surface maintain a preset angle relationship. The device spatial coordinates and lens orientation parameters of each target observation position are recorded as fixed device pose parameters, which include the device origin coordinates and optical axis direction vector; Based on the device's pose parameters, the 3D vision acquisition device is controlled to simultaneously acquire point cloud data and 2D image data at each observation position, forming a set of scene 3D vision data covering the distribution of surface features of the workpiece.

7. The precision workpiece positioning method based on three-dimensional vision of claim 3, wherein, The step of performing boundary detection processing on the two-dimensional image data in the fused data set to extract the boundary contour lines between the workpiece and the background in the image includes: Two-dimensional image data and corresponding point cloud data are spatially mapped to generate a fused image with depth information, wherein each pixel in the fused image is associated with its corresponding three-dimensional depth value. The fused image is subjected to multimodal feature enhancement processing, and the brightness gradient of the grayscale image and the depth gradient of the depth image are calculated simultaneously to generate a joint gradient response map. The boundary region between the workpiece and the background in the joint gradient response map shows a significant gradient change. Based on the joint gradient response map, boundary candidate regions are extracted, and a set of pixels whose gradient response values ​​exceed the local mean are identified as boundary candidate points through an adaptive threshold segmentation operation. The boundary candidate points are subjected to depth continuity verification processing, and pixels with depth value changes within the neighborhood that are less than a preset range are retained to form boundary candidate regions with consistent depth. The candidate boundary regions with consistent depth are subjected to closure checks. Continuous candidate boundary points are connected by a contour tracking algorithm to generate a closed workpiece boundary contour line. The contour line simultaneously satisfies the dual characteristics of image grayscale boundary and depth discontinuity boundary.

8. A computer system comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

Images