A rigid body pose prediction method and device based on a marker point linear mapping

Abstract

The application relates to a rigid body pose prediction method and device based on a marker point linear mapping, the method learns the spatial mapping relationship between multiple marker points on a rigid object, accurate pose prediction of the marker points is realized when part of the marker points are blocked, a plurality of sample images of different perspectives are collected, a homography matrix of each marker point relative to a reference coordinate system is calculated, and a linear mapping model between a first marker point and a second marker point is established through vectorization and a least square method; when a target marker point is blocked, the homography matrix of the visible first marker point is used to predict the homography matrix of the blocked marker point through the established linear mapping model, and then the pose of the blocked marker point is restored. The problem that measurement fails due to marker point blocking in a traditional visual measurement system is solved, and the robustness and environmental adaptability of the system are significantly improved.

Description

Technical Field

[0001] This application relates to the field of attitude prediction technology, and in particular to a rigid body pose prediction method and apparatus based on linear mapping of marker points. Background Technology

[0002] Visual marker-based pose estimation technology has wide applications in key scenarios such as structural health monitoring, robot autonomous navigation, and industrial automation measurement. Its core principle is to determine the three-dimensional position and orientation of a rigid object by analyzing the spatial distribution of marker points in an image. However, existing technologies face insurmountable bottlenecks: in real industrial environments, marker points are often partially or completely obscured due to factors such as equipment structure occlusion, environmental interference, or operator activity. This results in the system being unable to acquire complete marker point data, leading to interruptions or failures in the pose estimation process.

[0003] To improve the robustness of the system under occlusion conditions, traditional solutions often rely on deploying a large number of redundant markers. This approach not only significantly increases hardware procurement and installation costs but also makes the calibration process extremely cumbersome, while greatly increasing the resource consumption of subsequent image processing and computation. More importantly, existing methods generally adopt a passive observation mechanism, directly measuring only the currently visible markers and completely ignoring the inherent geometric constraints between multiple markers within a rigid structure. Therefore, they cannot reasonably infer the state of occluded markers. Although some studies have attempted to address the occlusion problem through multi-camera collaborative observation or simultaneous localization and mapping (SMR) techniques, multi-camera solutions require complex hardware synchronization and external parameter calibration processes. Meanwhile, SLAM methods based on environmental features are limited by the richness of scene textures. Under conditions where markers are completely occluded or environmental features are sparse, they still cannot provide continuous and reliable pose information, severely restricting the practicality and adaptability of the technology in complex industrial environments. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] The main objective of this disclosure is to propose a rigid body pose prediction method and apparatus based on linear mapping of marker points, which can solve the problem of measurement failure caused by marker point occlusion in traditional visual measurement systems, and significantly improve robustness and environmental adaptability.

[0006] A first aspect of this application provides a rigid body pose prediction method based on linear mapping of marker points, the method comprising: In response to a pose prediction instruction for a target marker point in a target image, a mapping model between multiple sample images and a reference image is determined; wherein the multiple sample images and the reference image are images of a rigid object captured at different shooting positions and angles; the rigid object is provided with multiple coded marker points, the multiple coded marker points include a first marker point and at least one second marker point, the target marker point is any one of the at least one second marker point, and the reference image contains the multiple marker points of the rigid object; The calculation method of the mapping model includes: Each homography matrix between each marker point in each sample image and a corresponding marker point in the reference image is determined to obtain multiple homography matrices. Each element in the homography matrix is ​​converted into a corresponding vector to obtain multiple vectors corresponding to the multiple homography matrices; Construct a first matrix from the vector values ​​associated with the first marker point in each of the vectors; Construct a corresponding second matrix from the vector values ​​associated with each second marker point in each of the vectors to obtain at least one second matrix corresponding to the at least one second marker point; Based on the first matrix and the at least one second matrix, construct the mapping model between each second marker point and the first marker point; Calculate the target homography matrix from the first marker point in the target image to the first marker point in the reference image, and convert the elements in the target homography matrix into target vectors, so as to solve the pose of the target marker point according to the mapping model and the target vectors.

[0007] The rigid body pose prediction method based on linear mapping of marker points provided in this embodiment has at least the following beneficial effects: This method acquires sample images from multiple viewpoints, then uses a homography matrix to establish a linear mapping model between marker points in different sample images and marker points in a reference image. When a target marker point is occluded, only the first visible marker point needs to be detected, and the pose of the occluded visual target marker point can be predicted using the mapping model. By establishing the mapping relationship between marker points, this method can predict the pose of other occluded marker points using a limited number of visible marker points, thereby reducing the dependence on redundant marker points. This helps to reduce hardware, calibration, and computational costs, while maintaining or even improving the accuracy and reliability of pose estimation.

[0008] A second aspect of this application provides a rigid body pose prediction device based on linear mapping of marker points, the device comprising: The instruction response module is used to determine a mapping model between multiple sample images and a reference image in response to a pose prediction instruction for a target marker point in a target image; wherein the multiple sample images and the reference image are images of a rigid object captured at different shooting positions and angles; the rigid object is provided with multiple coded marker points, the multiple coded marker points include a first marker point and at least one second marker point, the target marker point is any one of the at least one second marker point, and the reference image contains the multiple marker points of the rigid object; The calculation method of the mapping model includes: Each homography matrix between each marker point in each sample image and a corresponding marker point in the reference image is determined to obtain multiple homography matrices. Each element in the homography matrix is ​​converted into a corresponding vector to obtain multiple vectors corresponding to the multiple homography matrices; Construct a first matrix from the vector values ​​associated with the first marker point in each of the vectors; Construct a corresponding second matrix from the vector values ​​associated with each second marker point in each of the vectors to obtain at least one second matrix corresponding to the at least one second marker point; Based on the first matrix and the at least one second matrix, construct the mapping model between each second marker point and the first marker point; The pose confirmation module is used to calculate the target homography matrix from the first marker point in the target image to the first marker point in the reference image, and convert the elements in the target homography matrix into target vectors, so as to calculate the pose of the target marker point according to the mapping model and the target vectors.

[0009] A third aspect of this application provides an electronic device including at least one controller and a memory for communicatively connecting to the controller; the memory stores instructions executable by the at least one controller, the instructions being executed by the at least one controller to cause the at least one controller to perform a rigid body pose prediction method based on linear mapping of marker points as described above.

[0010] In a fourth aspect, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a rigid body pose prediction method based on linear mapping of marker points as described above.

[0011] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the rigid body pose prediction method based on linear mapping of marker points provided in this application embodiment; Figure 2 This is a schematic diagram of the rigid body pose prediction device based on linear mapping of marker points provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0015] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0016] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed or function in a specific orientation, and therefore should not be construed as a limitation of this application.

[0017] In visual marker-based pose estimation applications, the pose estimation process is interrupted and measurement fails when markers are partially or completely occluded. To improve system robustness, a large number of redundant markers need to be deployed, which increases the complexity of hardware configuration, calibration process, and computational cost. In complex industrial environments, occlusion occurs frequently, reducing system reliability. The essence of this is that existing technologies adopt a passive measurement mode and fail to utilize the spatial constraint relationship between multiple markers on a rigid body for state reasoning.

[0018] For example, in industrial measurement scenarios, when monitoring the pose of large mechanical equipment, operators or moving parts may temporarily obstruct the markers installed on the surface of the equipment. When obstruction occurs, traditional methods cannot obtain the coordinate information of the obstructed markers and thus terminate the pose calculation. The camera position needs to be readjusted or the markers need to be added, which interrupts the monitoring process, disrupts the data continuity, and affects the integrity of subsequent analysis processes.

[0019] If the above problems are not solved, the adaptability of the pose estimation system under occlusion conditions will be weakened, the frequency of measurement interruptions will be increased, the system maintenance requirements will be increased, the overall operating efficiency will be affected, and the implementation effect of related application areas will be restricted.

[0020] like Figure 1 As shown in one embodiment of this application, a rigid body pose prediction method based on linear mapping of marker points is provided. The method includes: Step S110: In response to the pose prediction instruction of the target marker point in the target image, determine the mapping model between multiple sample images and a reference image; wherein, the multiple sample images and the reference image are images of a rigid object taken at different shooting positions and angles; the rigid object is provided with multiple coded marker points, the multiple coded marker points include a first marker point and at least one second marker point, the target marker point is any one of the at least one second marker point, and the reference image contains multiple marker points of the rigid object; The calculation methods for the mapping model include: Determine the homography matrix between each marker point in each sample image and the corresponding marker point in the reference image to obtain multiple homography matrices; Each element in a homography matrix is ​​converted into a corresponding vector to obtain multiple vectors corresponding to multiple homography matrices; Construct a first matrix from the vector values ​​associated with the first marker point in each vector; Construct a corresponding second matrix from the vector values ​​associated with each second marker point in each vector, so as to obtain at least one second matrix corresponding to at least one second marker point; Based on the first matrix and at least one second matrix, construct a mapping model between each second marker point and the first marker point; Step S120: Calculate the target homography matrix from the first marker point in the target image to the first marker point in the reference image, and convert the elements in the target homography matrix into target vectors, so as to solve the pose of the target marker points according to the mapping model and the target vectors.

[0021] For ease of understanding, the following explains some key terms in this embodiment: A rigid object is an object whose distance between any two points within it remains constant during motion and the application of forces. In practical applications, many industrial parts and robot components can be considered rigid objects, whose geometry and dimensions remain fixed during pose estimation.

[0022] Visual markers (or visual reference points) are visual feature points with encoded information placed on the surface of a rigid object. These markers are easily detected and identified in images and are used to assist computer vision systems in object recognition, localization, and pose estimation. Examples include QR codes, circular dots, and square patterns.

[0023] Sample images refer to a series of images obtained by photographing a rigid object with marked points at different shooting positions and angles. These images are used to train or build models to learn the spatial relationships between the marked points from different perspectives.

[0024] The target image refers to the image to be predicted; it is also an image of a rigid object. By analyzing the marker points in the target image, its pose can be predicted.

[0025] A reference image is an image obtained by photographing a rigid object with marked points at a specific, known, or predetermined shooting position and angle. The reference image serves as a benchmark, used for comparison with sample images and to establish a mapping relationship between the marked points. A reference image typically contains all the marked points on the rigid object.

[0026] A marker mapping model is a mathematical model that describes the spatial mapping relationship between marker points in a sample image and their corresponding marker points in a reference image. Built by learning from a large amount of sample data, the model can capture the geometric transformations of a rigid body under different viewpoints.

[0027] A homography matrix is ​​a 3x3 nonsingular matrix used as a mathematical tool to describe perspective transformations between two planes. In computer vision, when a point on one plane is projected onto another, this projection relationship can be represented by a homography matrix. In pose estimation, a homography matrix can be used to describe the two-dimensional projection transformation of the same marker point in different images (e.g., sample and reference images).

[0028] Pose refers to the position and orientation of an object in three-dimensional space.

[0029] This method first responds to pose prediction commands from target markers in the target image. These commands can be triggered by an external system; for example, when pose monitoring of a specific rigid object is required, the system will issue a corresponding prediction command. Upon receiving the command, the system will initiate the pose prediction process.

[0030] Upon receiving the pose prediction command, a mapping model between multiple sample images and a reference image needs to be determined. This process is one of the core steps of this method. Multiple sample images are obtained by photographing a rigid object with multiple marker points at different shooting positions and angles. For example, a single camera can be used to photograph the rigid body at multiple preset or random positions to obtain images from different perspectives. The reference image is an image of the rigid body taken at a fixed and known shooting position and angle, containing all the marker points on the rigid body. The multiple marker points on the rigid object include a first marker point and at least one second marker point. The first marker point can be designated as a reference point on the rigid body.

[0031] The methods for determining the mapping model include: Step S210: Determine the homography matrix between each marker point in each sample image and the corresponding marker point in the reference image to obtain multiple homography matrices; Step S220: Convert the elements in each homography matrix into corresponding vectors to obtain multiple vectors corresponding to multiple homography matrices; Step S230: Construct a first matrix from the vector values ​​associated with the first marker point in each vector; Step S240: Construct a corresponding second matrix from the vector values ​​associated with each second marker point in each vector, so as to obtain at least one second matrix corresponding to at least one second marker point.

[0032] Step S250: Based on the first matrix and at least one second matrix, construct a mapping model between each second marker point and the first marker point.

[0033] A first matrix is ​​constructed based on the first marker points from multiple element vectors. The construction of the first matrix aims to collect homography information between the first marker points in all sample images and their corresponding first marker points in the reference image. Specifically, the vector corresponding to the first marker point in each sample image can be used as a column or row of the first matrix, thus forming a matrix containing the homography features of all first marker points. A second matrix is ​​constructed based on the second marker points from multiple element vectors. The construction of the second matrix is ​​similar to that of the first matrix, but it collects homography information between the second marker points in all sample images and their corresponding second marker points in the reference image. The vector corresponding to the second marker point in each sample image is used as a column or row of the second matrix. The second matrix and the first matrix are used together to establish the mapping relationship between them.

[0034] Based on the first and second matrices, a mapping model is calculated. This application requires establishing at least one mapping relationship between the second marker point and the first marker point. For example, each sample image contains marker points 0, 1, 2, 3, and 4, where 0 is the first marker point. A linear mapping relationship (y≈ax+b) needs to be established between 0 and 1, 0 and 2, 0 and 3, and 0 and 4. The calculation aims to find a mathematical model that can map the data in the first matrix (representing the transformation of the first marker point) to the data in the second matrix (representing the transformation of the second marker point). This allows the model to optimally describe the correlation between the homography transformations of the first and second marker points under different viewpoints.

[0035] After the mapping model is constructed, the target homography matrix from the first marker point in the target image to the first marker point in the reference image needs to be calculated. Then, based on the constructed mapping model and the calculated target homography matrix, the pose of the target marker point is calculated. For example, by inputting the target vector corresponding to the target homography matrix into the mapping model, the model can infer the vector of the target marker point in the target image relative to the reference image based on the learned mapping relationship. Once the target vector of the target marker point relative to the reference image is obtained, it can be further converted into pose information in three-dimensional space.

[0036] In actual production, when it is necessary to predict the pose of a tool, the system acquires a target image. Assume that in the target image, the first marker point is clearly visible, but a crucial second marker point cannot be directly observed due to occlusion by the robotic arm itself. In this case, the system first calculates the target homography matrix from the first marker point in the target image to the first marker point in the reference image. This target homography matrix reflects the reference transformation of the current target image relative to the reference image.

[0037] Subsequently, the system inputs the vector corresponding to this target homography matrix into a pre-built mapping model. Since the model has already learned the mapping relationship between the first and second marker points, even if the second marker point in the target image is occluded, the model can still infer and calculate the vector from the second marker point in the target image to the corresponding second marker point in the reference image based on the input target homography matrix. Once this inferred homography matrix is ​​obtained, the system can convert it into the precise pose of the second marker point in 3D space.

[0038] The method provided in this embodiment has at least the following beneficial effects: This method acquires sample images from multiple viewpoints, then uses a homography matrix to establish a linear mapping model between marker points in different sample images and marker points in a reference image. When a target marker point is occluded, only the first visible marker point needs to be detected, and the pose of the occluded visual target marker point can be predicted using the mapping model. By establishing the mapping relationship between marker points, this method can predict the pose of other occluded marker points using a limited number of visible marker points, thereby reducing the dependence on redundant marker points. This helps to reduce hardware, calibration, and computational costs, while maintaining or even improving the accuracy and reliability of pose estimation.

[0039] In some embodiments of this application, step S110, which converts each element in a homography matrix into a corresponding vector to obtain multiple vectors corresponding to multiple homography matrices, includes: Step S310: Normalize the homography matrix between each marker point in each sample image and the corresponding marker point in the reference image to obtain multiple normalized homography matrices. Step S320: Extract the first 8 elements from each normalized homography matrix, construct a corresponding initial vector and transpose it to obtain multiple vectors; wherein, the first 8 elements include all elements except the element in the 3rd row and 3rd column of the normalized homography matrix.

[0040] The normalization operation aims to eliminate the scale uncertainty of the homography matrix, ensuring that all homography matrices are compared and processed at a uniform scale, thereby improving the numerical stability and consistency of subsequent calculations.

[0041] The first normalization method is to normalize the last element of the homography matrix to 1, that is, to divide all elements of the matrix by the last element. The second normalization method is to normalize the Frobenius norm of the homography matrix to 1, that is, to divide all elements of the matrix by their Frobenius norm. Normalization ensures the comparability of homography matrices calculated between different sample images and marker points, laying the foundation for subsequent mapping model construction. This embodiment uses the first method.

[0042] The first 8 elements of each normalized homography matrix are extracted to construct a corresponding element vector, resulting in multiple element vectors for multiple normalized homography matrices. The first 8 elements are all elements except the element in the 3rd row and 3rd column of the normalized homography matrix. A homography matrix is ​​a 3x3 matrix, but since the last element is normalized to 1, it actually only has 8 independent degrees of freedom. Extracting the first 8 elements and converting them into an 8-dimensional element vector is to transform matrix data into vector form, facilitating linear algebra operations and the construction of linear mapping models. This transformation represents each homography matrix as a vector of uniform dimension, allowing for comparison and correlation between different homography matrices through vector operations.

[0043] This embodiment normalizes the homography matrix determined from each marker point in each sample image to the corresponding marker point in the reference image, ensuring that all homography matrices are processed at a uniform scale. This eliminates scale uncertainty and improves the numerical stability and consistency of subsequent calculations. Subsequently, the first eight independent elements are extracted from each normalized homography matrix and constructed into an element vector. This vectorization process converts the matrix-form homography data into a vector form of uniform dimension, providing standardized input for subsequent linear algebra operations. This systematic data processing and model construction approach enables the mapping model to more accurately and robustly capture the intrinsic correlation of pose transformations between different marker points on a rigid object, thus providing a solid foundation for subsequent target marker point pose calculation.

[0044] In some embodiments of this application, step S250, which involves constructing a mapping model based on the first matrix and the second matrix, includes: Step S410: Solve the mapping model between the first matrix and the second matrix based on the least squares method.

[0045] Least squares is a commonly used mathematical optimization technique. Its core idea is to find a best-fit function that minimizes the sum of squared residuals from all data points to that function. By applying least squares, noise and errors in the data can be effectively handled, resulting in a robust and accurate model. When solving the mapping model between the first and second matrices, least squares can fit the mathematical expression that best represents the linear or nonlinear relationship between these data points from a matrix containing multiple sample data points. For example, linear least squares can be used to represent the mapping model as a system of linear equations, and the model parameters can be determined by solving the system of equations. Alternatively, weighted least squares can be used, assigning different weights based on the reliability of different data points to further improve the model's accuracy and robustness.

[0046] In some embodiments of this application, the process of determining the homography matrix between the marker points in the sample image and the corresponding marker points in the reference image in step S110 includes: Step S510: Extract the coordinates of the first corner point of the marked point in the sample image.

[0047] Step S520: Extract the coordinates of the second corner point of the marked point in the reference image.

[0048] Step S530: Solve the linear equation system based on the points, and calculate the homography matrix using the coordinates of the first and second corner points.

[0049] Specifically, extracting the first corner coordinates of the marker points in the sample image refers to identifying and locating the key feature points of the marker points in the sample image and obtaining the precise position information of these feature points in the image coordinate system. These corner coordinates are the basic data for subsequent calculation of the homography matrix. Similarly, extracting the second corner coordinates of the marker points in the reference image refers to identifying and locating the key feature points of the marker points corresponding to the marker points in the sample image in the reference image and obtaining the precise position information of these feature points in the image coordinate system. Solving a system of linear equations based on points to calculate the homography matrix between the first and second corner coordinates involves using known pairs of corresponding points to solve for a 3x3 transformation matrix that describes the perspective transformation relationship from one plane to another. For example, the Direct Linear Transformation (DLT) algorithm can be used to solve for the homography matrix. The DLT algorithm transforms the nonlinear equations of the homography matrix into a linear equation system and uses at least four pairs of non-collinear corresponding points to solve for the eight unknown parameters of the matrix.

[0050] This embodiment ensures that the homography matrix of each marker point across different images is accurately and robustly calculated before constructing the marker mapping model. This significantly improves the accuracy of the homography matrix, thereby providing high-quality input data for the subsequent marker mapping model constructed based on these homography matrices. Ultimately, this helps improve the overall accuracy and reliability of the rigid body pose prediction method, enabling more accurate target marker pose calculation when responding to pose prediction commands.

[0051] For ease of understanding, this application also provides an embodiment of a rigid body pose prediction method based on linear mapping of marker points, including the following: Teaching phase: Step S910, arrange on the rigid object Marker points , Each marker point contains at least one first marker point and at least one second marker point; a reference image containing all marker points is acquired by a camera, all marker points in the reference image are detected, and the identifier ID and corner coordinates of each marker point are recorded.

[0052] The markers are coded planar markers such as ArUco codes, AprilTags, or QR codes.

[0053] Step S920: Keeping the rigid object and the markers on it stationary, move the camera to different positions and angles to collect data. ( Given at least 20 sample images, detect the IDs of all marker points and the coordinates of corner points in the sample images.

[0054] Step S930: Based on the detected marker points in the sample image, calculate the homography matrix from each marker point in the sample image to the corresponding marker point in the reference image. Among them, homography matrix It is a 3x3 matrix. The homography matrix is ​​calculated based on the perspective transformation of the corner coordinates of the marked points, specifically as follows: Extract the corner coordinates Corners1 of any marker point in the baseline image; Extract the corner coordinates Corners2 of any marker point in the sample image; Solve the system of linear equations based on the corresponding points, calculate the mapping relationship from corner coordinates Corners2 to corner coordinates Corners1, and thus obtain the homography matrix. The formula is expressed as follows: ; in, The homogeneous coordinates of the corner points on the reference image (2D points are represented by 3D vectors, with the last dimension being 1); These are the homogeneous coordinates of the corner points on the sample image corresponding to those on the reference image.

[0055] For example, ; To verify the validity of the homography matrix, calculate its determinant. .

[0056] when When the homography matrix is ​​determined to be degenerate or close to singular, it is marked as invalid.

[0057] When | When determining the validity of the homography matrix, The preset threshold can be set to 1. .

[0058] Step S940: Utilizing the scale invariance of the homography matrix, the homography matrices are... After normalization, we get .For example, ; Step S950: The normalized homography matrices are... Transform a 3x3 matrix into a vector containing 9 elements. Normalized homography matrix The element in row 3 and column 3 is 1, so we take the vector. The first 8 parameters are combined and transposed to form the transpose vector. .For example, ; ; Step S960, based on each transpose vector Constructing the input matrix from visual reference markers ,For example, ; Step S970, based on each transpose vector Constructing the output matrix from visual target markers .For example: ; ; ; Step S980: Solve the input matrix using the least squares method. With output matrix The linear mapping model.

[0059] The linear mapping model is solved using the least squares method, specifically through an augmented matrix: Transform the linear mapping relationship into a least squares optimization problem: ; ; ; The linear relationship is expressed as , input matrix With output matrix Substituting the values ​​into the mapping relationship, we can construct: ; This represents the Frobenius norm, which is the sum of the squares of all elements in the matrix.

[0060] Eliminate offset Construct the augmented matrix: , Substituting into the above formula transforms the problem into a solution: ; Finally, the required parameters are solved. The results are as follows: ; Extract the first 8 columns of the 8 rows as the transformation matrix. The result is that the 8th row and 9th column are extracted as the offset vector. The result.

[0061] Model quality assessment first involves calculating the predicted output matrix using linear mapping relationships. Then, by predicting the output matrix and output matrix Calculate the residual matrix Then use the residual matrix Calculate the sum of squares of the residual matrix Sum of squares of the output matrix Finally, based on the sum of squares of the residual matrix and the total sum of squares of the output matrix Determine the coefficient of determination , The range of values ​​is According to the coefficient of determination To evaluate the quality of a linear mapping relationship, when If the value is greater than 0.9, the linear mapping model is considered valid. If not, the system will prompt the user to add more sample images or restart the linear mapping model.

[0062] The specific calculation methods for model quality assessment include: Calculate the predicted output matrix : ; in, for A row vector of all 1s; output matrix With the output prediction matrix Calculate the residual matrix: ; in, 8 lines For a matrix of columns, calculate the sum of squared residuals: ; in, , , Represents the residual matrix Sum the squares of all elements; calculate the output matrix. row mean vector : ; in, , , Given the sample size; calculate the total sum of squares. : ; in, , Through the total sum of squares With calculation of residual sum of squares Calculate the coefficient of determination : ; according to Values ​​used to determine model quality: when A value ≥ 0.90 is considered valid. when If the value is less than 0.90, it is considered poor, and suggestions are made to increase the sample size or execute the first phase.

[0063] To ensure the reliability and accuracy of the model, a model validity evaluation mechanism is further introduced here, by setting a reasonable... The system can automatically determine the validity of the currently constructed marker point mapping model by setting a threshold. This evaluation mechanism ensures that only validated, high-precision mapping models are used for subsequent target marker point pose calculations, thereby significantly improving the accuracy and robustness of the entire rigid body pose prediction method and avoiding prediction errors caused by poor model quality. This ensures that the mapping model used for rigid body pose prediction has high accuracy and reliability. After model construction, the system can quantitatively evaluate the model's fit to the actual data and, based on... The evaluation process determines whether the model meets the expected accuracy requirements, which avoids using low-quality or inaccurate models for pose prediction, thereby significantly reducing prediction errors and improving the accuracy and stability of target marker pose calculation. This evaluation process provides quality assurance for subsequent pose prediction, making the entire rigid body pose prediction method more robust and practical.

[0064] Prediction phase; In step S990, when the visual target marker is occluded, firstly, the visual reference marker is detected, and the homography matrix of the visual reference marker on the reference image is calculated. Then the homography matrix Vectorize to transpose Finally, the transpose vector Substituting the linear mapping relationship into the solution, the target marker point vector is obtained. and the target marker point vector Convert to the pose of the target marker point.

[0065] Ensure augmented matrix: The determinant of the matrix is ​​greater than 10-10, and the matrix is ​​invertible.

[0066] The method also includes numerical stability checks: Calculate the condition number of a matrix:

[0067] when When the value is unstable, use [the appropriate method]. Decompose and solve.

[0068] The method provided in this embodiment has the following beneficial effects: This method learns the spatial mapping relationship between multiple marker points on a rigid object to accurately predict the pose of partially occluded marker points. It includes a teaching phase and a prediction phase: In the teaching phase, sample images from multiple different viewpoints are acquired, and the homography matrix of each marker point relative to the reference coordinate system is calculated. A linear mapping model between the first and second marker points is established using vectorization and the least squares method. In the prediction phase, when the target marker point is occluded, the homography matrix of the occluded marker point is predicted using the homography matrix of the visible reference marker point and the established linear mapping model, thus recovering its pose. This method learns the spatial mapping relationship between multiple marker points on a rigid body during the teaching phase. In the prediction phase, when the target marker point is occluded, only the visible reference marker point needs to be detected to predict the pose of the occluded marker point using the established linear mapping model. This solves the problem of measurement failure caused by marker point occlusion in traditional visual measurement systems, significantly improving the robustness and environmental adaptability of the system.

[0069] like Figure 2 As shown in one embodiment of this application, a rigid body pose prediction device based on linear mapping of marker points is provided. The device includes: The instruction response module 1100 is used to determine a mapping model between multiple sample images and a reference image in response to a pose prediction instruction for a target marker point in a target image; wherein, the multiple sample images and the reference image are images of a rigid object taken at different shooting positions and angles; the rigid object is provided with multiple coded marker points, the multiple coded marker points include a first marker point and at least one second marker point, the target marker point is any one of the at least one second marker point, and the reference image contains multiple marker points of the rigid object; The calculation methods for the mapping model include: Determine the homography matrix between each marker point in each sample image and the corresponding marker point in the reference image to obtain multiple homography matrices; Each element in a homography matrix is ​​converted into a corresponding vector to obtain multiple vectors corresponding to multiple homography matrices; Construct a first matrix from the vector values ​​associated with the first marker point in each vector; Construct a corresponding second matrix from the vector values ​​associated with each second marker point in each vector, so as to obtain at least one second matrix corresponding to at least one second marker point; Based on the first matrix and at least one second matrix, construct a mapping model between each second marker point and the first marker point; The pose confirmation module 1200 is used to calculate the target homography matrix from the first marker point in the target image to the first marker point in the reference image, and convert the elements in the target homography matrix into target vectors, so as to solve the pose of the target marker point according to the mapping model and the target vectors.

[0070] It should be noted that the rigid body pose prediction device based on linear mapping of marker points provided in this embodiment is based on the same inventive concept as the rigid body pose prediction method based on linear mapping of marker points described above. Therefore, the content of the rigid body pose prediction method based on linear mapping of marker points described above is also applicable to the content of the rigid body pose prediction device based on linear mapping of marker points in this embodiment, and will not be repeated here.

[0071] like Figure 3 An embodiment of this application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described rigid body pose prediction method based on linear mapping of marker points. The electronic device includes: At least one battery; At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the rigid body pose prediction method based on linear mapping of marker points described above in this disclosure.

[0072] Electronic devices can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0073] The electronic devices according to embodiments of this application will now be described in detail.

[0074] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to perform a rigid body pose prediction method based on linear mapping of marker points according to an embodiment of this disclosure.

[0075] The input / output interface 1800 is used to implement information input and output. The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900); The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0076] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described rigid body pose prediction method based on linear mapping of marker points.

[0077] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0078] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0079] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0082] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A rigid body pose prediction method based on linear mapping of marker points, characterized in that, The method includes: In response to a pose prediction instruction for a target marker point in a target image, a mapping model between multiple sample images and a reference image is determined; wherein the multiple sample images and the reference image are images of a rigid object captured at different shooting positions and angles; the rigid object is provided with multiple coded marker points, the multiple coded marker points include a first marker point and at least one second marker point, the target marker point is any one of the at least one second marker point, and the reference image contains the multiple marker points of the rigid object; The calculation method of the mapping model includes: Each homography matrix between each marker point in each sample image and a corresponding marker point in the reference image is determined to obtain multiple homography matrices. Each element in the homography matrix is ​​converted into a corresponding vector to obtain multiple vectors corresponding to the multiple homography matrices; Construct a first matrix from the vector values ​​associated with the first marker point in each of the vectors; Construct a corresponding second matrix from the vector values ​​associated with each second marker point in each of the vectors to obtain at least one second matrix corresponding to the at least one second marker point; Based on the first matrix and the at least one second matrix, construct the mapping model between each second marker point and the first marker point; Calculate the target homography matrix from the first marker point in the target image to the first marker point in the reference image, and convert the elements in the target homography matrix into target vectors, so as to solve the pose of the target marker point according to the mapping model and the target vectors.

2. The rigid body pose prediction method based on linear mapping of marker points according to claim 1, characterized in that, The step of constructing the mapping model between each second marker point and the first marker point based on the first matrix and the at least one second matrix includes: Based on the first matrix and the at least one second matrix, a mapping model between each second marker point and the first marker point is constructed using the least squares method.

3. The rigid body pose prediction method based on linear mapping of marker points according to claim 2, characterized in that, The step of converting each element in the homography matrix into a corresponding vector to obtain multiple vectors corresponding to the multiple homography matrices includes: The homography matrix between each marker point in each of the sample images and the corresponding marker point in the reference image is normalized to obtain multiple normalized homography matrices. Extract the first 8 elements from each of the normalized homography matrices, construct a corresponding initial vector, and transpose it to obtain the plurality of vectors; wherein, the first 8 elements include all elements except the element in the 3rd row and 3rd column of the normalized homography matrix.

4. The rigid body pose prediction method based on linear mapping of marker points according to claim 2, characterized in that, The process of determining the homography matrix between the marker points in the sample image and the corresponding marker points in the reference image includes: Extract the coordinates of the first corner point of the marked point in the sample image; Extract the coordinates of the second corner point of the marked point in the reference image; Solve the linear equations based on the points, and calculate the homography matrix using the coordinates of the first and second corner points.

5. The rigid body pose prediction method based on linear mapping of marker points according to claim 4, characterized in that, After determining the homography matrix between each marker point in each of the sample images and a corresponding marker point in the reference image, the method further includes: Calculate the determinant of the homography matrix; If the determinant is greater than a preset threshold, the homography matrix is ​​determined to be valid. The step of converting each element in the homography matrix into a corresponding vector to obtain multiple vectors corresponding to the multiple homography matrices includes: Each element in a valid homography matrix is ​​converted into a corresponding vector to obtain multiple vectors corresponding to multiple valid homography matrices.

6. The rigid body pose prediction method based on linear mapping of marker points according to claim 2, characterized in that, After determining the mapping model between the multiple sample images and the reference image, the method further includes: Calculate the third matrix of prediction : ； in, For the first matrix, Let be the change matrix of the weight coefficients. For bias terms, The total number of sample images. for A row vector of all 1s; According to the third matrix And the second matrix, calculate the residual matrix. : ； in, This is the second matrix; According to the residual matrix ,calculate value: ； in, The residual matrix The first in OK Column elements; Calculate the row mean vector based on the second matrix. : ； in, For the second matrix The first in OK Column elements; According to the second matrix and the row mean vector ,calculate value: ； in, ; According to the above Value and the stated Value, calculation value: ； In the If the value exceeds a preset threshold, the mapping model is deemed valid. The step of calculating the pose of the target marker point based on the mapping model and the target vector includes: The pose of the target marker point is calculated based on the effective mapping model and the target vector.

7. The rigid body pose prediction method based on linear mapping of marker points according to claim 2, characterized in that, The marker point is an ArUco code, AprilTag, or QR code.

8. A rigid body pose prediction device based on linear mapping of marker points, characterized in that, The device includes: The instruction response module is used to determine a mapping model between multiple sample images and a reference image in response to a pose prediction instruction for a target marker point in a target image; wherein the multiple sample images and the reference image are images of a rigid object captured at different shooting positions and angles; the rigid object is provided with multiple coded marker points, the multiple coded marker points include a first marker point and at least one second marker point, the target marker point is any one of the at least one second marker point, and the reference image contains the multiple marker points of the rigid object; The calculation method of the mapping model includes: Each homography matrix between each marker point in each sample image and a corresponding marker point in the reference image is determined to obtain multiple homography matrices. Each element in the homography matrix is ​​converted into a corresponding vector to obtain multiple vectors corresponding to the multiple homography matrices; Construct a first matrix from the vector values ​​associated with the first marker point in each of the vectors; Construct a corresponding second matrix from the vector values ​​associated with each second marker point in each of the vectors to obtain at least one second matrix corresponding to the at least one second marker point; Based on the first matrix and the at least one second matrix, construct the mapping model between each second marker point and the first marker point; The pose confirmation module is used to calculate the target homography matrix from the first marker point in the target image to the first marker point in the reference image, and convert the elements in the target homography matrix into target vectors, so as to calculate the pose of the target marker point according to the mapping model and the target vectors.

9. An electronic device, characterized in that, It includes at least one controller and a memory for communicatively connecting with the controller; the memory stores instructions executable by the at least one controller, which, when executed by the at least one controller, causes the at least one controller to perform a rigid body pose prediction method based on linear mapping of marker points as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform a rigid body pose prediction method based on linear mapping of marker points as described in any one of claims 1 to 7.

Images