Moire-based pose estimation system and method suitable for generalized perspective angles

Abstract

The application discloses a moire-based pose estimation system and method suitable for generalizing perspective angles, which is applied to a camera device in a camera-based visual marker pose detection scene, and comprises a moire mixed feature separation module, a generalization moire feature representation module and a fine-grained feature point reconstruction module; the application provides generalization expression for any moire, and can also realize effective feature extraction and reasoning for the curved deformation moire caused by a large perspective angle between the camera and the visual marker. The application realizes sub-pixel level feature extraction of the visual marker, and provides higher six-degree-of-freedom pose estimation precision compared with traditional pixel level feature-based pose estimation.

Description

Technical Field

[0001] This invention belongs to the field of pose estimation technology, specifically relating to a pose estimation system and method based on moiré patterns applicable to generalized perspective angles. Background Technology

[0002] Ultra-high precision motion perception using computer vision is a key technology for many high-precision AR / VR applications, such as precision industrial manufacturing and image-guided surgery in medical fields. In recent years, moiré patterns, due to their high sensitivity to six-degree-of-freedom (6-DoF) pose changes, have become a new technology for achieving ultra-high precision motion pose perception. However, moiré pattern-based perception technology is still in its early stages; existing technologies can only process regular moiré pattern features and lack a universally applicable moiré pattern pose estimation method.

[0003] Existing moiré pose estimation techniques mainly include those based on customized moiré schemes and those based on color filter arrays (CFAs):

[0004] The basic idea behind customized moiré pattern schemes is to design a two-layer device to generate artificial moiré patterns. To ensure that the moiré pattern generated by the superposition of two grids changes with the camera's position and orientation, existing customized moiré pattern schemes typically use a two-layer device to ensure a certain spacing between the two grid layers. Thus, according to the camera imaging principle, when the camera moves a fixed distance, the two grid layers move different distances on the camera's imaging plane due to their different distances from the camera's optical center; that is, the two grid layers undergo relative movement, causing changes in the moiré image. These changes in moiré pattern characteristics are then used to further estimate changes in camera pose. However, existing two-layer devices mainly include the following types: periodic lens superposition printed grids, 3D printed perforated grids superimposed on electronic screen grids, and grids printed on both sides of a transparent wafer with a certain thickness. These methods all require complex customized two-layer equipment, making them difficult to deploy and apply conveniently and widely.

[0005] CFA-based solutions utilize the CFA embedded within the camera lens as one layer of the necessary double-layer grid for generating moiré patterns. However, existing CFA-based pose-aware solutions are limited to specific camera-screen interaction scenarios, and the CFA plane needs to be as parallel as possible to the interactive screen to ensure the effectiveness of the moiré features, i.e., the moiré pattern presents a periodic, regular grid. When there is a large perspective angle between the camera and the screen being photographed, the moiré pattern bends and deforms, and such methods cannot effectively extract moiré features and perform pose estimation. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, the present invention aims to provide a pose estimation system and method based on moiré patterns that is applicable to generalized perspective angles, thereby solving the problems of weak generalization and insufficient usability of existing pose estimation methods.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] This invention discloses a pose estimation system based on moiré patterns applicable to generalized perspective angles, applied in camera devices for pose detection scenarios based on visual markers captured by a camera. The system includes: a moiré pattern hybrid feature separation module, a generalized moiré pattern feature representation module, and a fine-grained feature point reconstruction module.

[0009] The moiré pattern hybrid feature separation module is used to extract and separate features from the hybrid moiré pattern image obtained when the camera captures visual markers;

[0010] The generalized moiré feature representation module is used to provide feature representation for moiré patterns after feature separation at any perspective angle, including periodic contour functions and curvature functions, so as to effectively characterize moiré features at generalized perspective angles.

[0011] The fine-grained feature point reconstruction module is used to perform super-resolution reconstruction of visual markers and extract sub-pixel level feature points. The sub-pixel level feature points of the visual markers reconstructed from each frame of the image are used as input to the PnP pose estimation algorithm to calculate the six-degree-of-freedom pose of the camera relative to the visual markers.

[0012] Furthermore, the camera device includes: a camera module with image acquisition and data transmission functions, and a mobile device with the camera module.

[0013] Furthermore, the visual marker is used to generate moiré patterns in the visual marker area of ​​the image when the camera captures the visual marker; the visual marker is a two-dimensional planar marker composed of one or more sets of periodic stripes with different directions, including a two-dimensional planar marker composed of two sets of periodic stripes with orthogonal directions; wherein each set of periodic stripes is composed of two linear stripes with high contrast colors, including black and white alternating linear stripes composed of black and white, and a pair of black and white stripes constitutes one cycle of the visual marker; the size and cycle of the visual marker are set according to the interaction distance requirements of different applications.

[0014] Furthermore, the hybrid moiré image refers to a moiré image in which two sets of periodic stripes overlap, wherein each set of periodic stripes is straight or curved.

[0015] Furthermore, the feature extraction in the moiré pattern hybrid feature separation module specifically includes:

[0016] The quadrilateral detection algorithm is used to extract the original region of interest where the moiré pattern is located in each frame of the image, and the coordinates of the four vertices of the quadrilateral are recorded.

[0017] A regular rectangle is extracted from the inside of the quadrilateral as the final moiré pattern target area. The color channel that makes the contrast of the moiré pattern stripes the strongest is selected to obtain a grayscale image of the moiré pattern target area. This grayscale image contains two mixed sets of moiré patterns.

[0018] Furthermore, the feature separation in the moiré pattern hybrid feature separation module specifically includes:

[0019] The grayscale image of the moiré target region is divided into M×N blocks, and a 2D FFT operation is performed on each block to obtain the corresponding spectrogram.

[0020] Extract two sets of frequency vectors from the spectrum, including the magnitude and direction of the frequency vectors; wherein, the two sets of frequency vectors represent the frequency characteristics of the two sets of moiré patterns mixed in the block; the magnitude of the frequency vectors represents the magnitude of the spatial frequency of the moiré pattern; the direction of the frequency vectors represents the direction of the spatial frequency of the moiré pattern.

[0021] Gabor filters are constructed based on the magnitude and direction of the extracted frequency vectors. Two different Gabor filters are constructed for the two sets of frequency vectors extracted in each block. Each block is filtered using the two constructed Gabor filters to obtain two sets of filtered fringes.

[0022] An iterative method based on relaxation labels is used to classify the two sets of stripes obtained after filtering for each block, and the two sets of moiré patterns contained in the grayscale image of the moiré pattern target region are separated into two images, each containing only one set of moiré patterns.

[0023] Furthermore, the generalized moiré feature representation module provides feature representation for moiré patterns after feature separation at any perspective angle, referring to the use of a periodic contour function p to represent moiré patterns with arbitrary curved shapes. m (x′) and the curvature function g m The moiré pattern m(x, y) is represented in a composite form as follows, where (x, y) represents the pixel coordinates of the image:

[0024] m(x, y) = p m (g m (x, y)).

[0025] Furthermore, the periodic contour function p m The method for calculating (x′) is as follows:

[0026]

[0027] Furthermore, the bending function g m The calculation process for (x, y) includes:

[0028] Perform a Hilbert transform on each row m(t) or column m(t) of each group of separated moiré patterns to obtain an analytic signal H(m)(t);

[0029] Extract the instantaneous angle of the analytical signal H(m)(t) to obtain the phase information of each row m(t) or column m(t);

[0030] The phase information of all rows or columns of the moiré pattern image is stitched together, and the resulting phase image is unwrapped to eliminate discontinuous 2π transitions in the phase image, resulting in a continuous phase information image.

[0031] Dividing the entire continuous phase information image by 2π yields the curvature function g. m (x, y).

[0032] Furthermore, the process of extracting sub-pixel level feature points in the fine-grained feature point reconstruction module includes:

[0033] According to the bending function g of the moiré pattern m The bending function g of (x, y) and camera color filter array (CFA) c Calculate the curvature function g of the visual marker projected onto the camera color filter array plane at (x, y). p (x, y), the calculation method is as follows:

[0034] g p (x, y) = g c (x, y)-g m (x, y)

[0035] Among them, the curvature function g of the two dimensions of the color filter array (horizontal stripes and vertical stripes) c (x, y) represent g respectively. c (x, y) = y and g c (x, y) = x;

[0036] The bending function g of visual markers p A new bending function g′ is obtained by bilinear upsampling (x, y). p (x, y), thereby improving the bending function from pixel-level resolution to sub-pixel-level resolution;

[0037] For the new bending function g′ p (x, y) performs a two-dimensional fitting operation, extending its range from the grayscale image region of the moiré target area to the original region of interest (ROI) where the moiré is located;

[0038] The expanded new bending function g′ p (x, y) and periodic contour function By performing composite processing, super-resolution visual labels can be obtained;

[0039] In the super-resolution visual marker image, the coordinates of the vertex closest to the four vertices of the original region of interest where the moiré pattern is located are selected as the sub-pixel level feature points of the reconstructed visual marker. These sub-pixel level feature points are used as the input of the PnP pose estimation algorithm to obtain the six-degree-of-freedom pose of the camera relative to the visual marker.

[0040] This invention also provides a pose estimation method based on moiré patterns applicable to generalized perspective angles, which, based on the above system, includes the following steps:

[0041] 1) Use a camera to capture video of visual markers in a pose detection scene based on camera-captured visual markers at a fixed frame rate. The visual marker regions in the video frames contain moiré patterns.

[0042] 2) Extract the original region of interest where the moiré pattern is located for each frame of the image and further crop out the target region of the moiré pattern with a regular rectangular shape;

[0043] 3) Separate the features of the two dimensions of the mixed moiré pattern image in the target region;

[0044] 4) Calculate the bending function for the moiré pattern features of the two separated dimensions respectively;

[0045] 5) Calculate the bending function of the visual marker based on the bending function of the moiré pattern and the bending function of the camera color filter array, and combine it with the periodic contour function to reconstruct the sub-pixel level feature points of the visual marker.

[0046] 6) Based on the sub-pixel level feature points of the visual markers reconstructed from each frame of the image, a pose estimation algorithm is used to output the six-degree-of-freedom pose of the camera relative to the visual markers.

[0047] The beneficial effects of this invention are:

[0048] 1. Generalization: This invention has generalization for the perspective angle between the camera and the visual marker; this invention provides a generalized expression for any moiré pattern, and can also achieve effective feature extraction and inference for the bending deformation moiré pattern caused by a large perspective angle between the camera and the visual marker.

[0049] 2. High precision: This invention achieves sub-pixel level feature extraction of visual markers, providing higher six-free pose estimation accuracy compared to pose estimation based on traditional pixel-level feature points.

[0050] 3. High availability and easy deployment: This invention differs from existing methods for generating regular moiré patterns based on dual-layer customized equipment. It can perform high-precision pose estimation based on simple single-layer visual markers, without the need for complex customized equipment, and is highly available and easy to deploy. Attached Figure Description

[0051] Figure 1 This is an architecture diagram of the system of the present invention in an embodiment;

[0052] Figure 2 This is a schematic diagram illustrating the principle of moiré pattern feature separation in this invention.

[0053] Figure 3 This is a schematic diagram illustrating the principle of bending function calculation in this invention;

[0054] Figure 4 This is a schematic diagram illustrating the principle of visual marker super-resolution in this invention;

[0055] Figure 5 This is a schematic diagram illustrating the principle of sub-pixel level feature point extraction in this invention. Detailed Implementation

[0056] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0057] Reference Figures 1 to 5 As shown, the present invention provides a pose estimation system based on moiré patterns applicable to generalized perspective angles, applied in camera devices in pose detection scenarios based on visual markers captured by a camera. The system includes: a moiré pattern hybrid feature separation module, a generalized moiré pattern feature representation module, and a fine-grained feature point reconstruction module.

[0058] The moiré pattern hybrid feature separation module is used to extract and separate features from the hybrid moiré pattern image obtained when the camera captures visual markers;

[0059] The camera device includes: a camera module with image acquisition and data transmission functions, and a mobile device with a camera module, such as a smartphone, smartwatch, smart glasses, head-mounted display device, smart handle, mobile robot, smart robotic arm device, etc.

[0060] The visual marker is used to generate moiré patterns in the visual marker area of ​​the image when the camera captures the visual marker; the visual marker is a two-dimensional planar marker composed of one or more sets of periodic stripes with different directions, including a two-dimensional planar marker composed of two sets of periodic stripes with orthogonal directions; each set of periodic stripes is composed of two linear stripes with high contrast colors, including black and white alternating linear stripes composed of black and white, and a pair of black and white stripes constitutes one cycle of the visual marker; the size and cycle of the visual marker are set according to the interaction distance requirements of different applications.

[0061] The mixed moiré pattern image refers to a moiré pattern image in which two sets of periodic stripes overlap, wherein each set of periodic stripes is straight or curved.

[0062] Specifically, the feature extraction in the moiré pattern hybrid feature separation module includes:

[0063] The quadrilateral detection algorithm is used to extract the original region of interest (ROI) where the moiré pattern is located in each frame of the image, and the coordinates of the four vertices of the quadrilateral are recorded.

[0064] A regular rectangle is extracted from the inside of the quadrilateral as the final moiré pattern target area. The color channel that makes the contrast of the moiré pattern stripes the strongest is selected to obtain a grayscale image of the moiré pattern target area. This grayscale image contains two mixed sets of moiré patterns.

[0065] Specifically, the feature separation in the moiré pattern hybrid feature separation module includes:

[0066] The grayscale image of the moiré target region is divided into M×N blocks, and a 2D FFT operation is performed on each block to obtain the corresponding spectrogram.

[0067] Extract two sets of frequency vectors from the spectrum, including the magnitude and direction of the frequency vectors; wherein, the two sets of frequency vectors represent the frequency characteristics of the two sets of moiré patterns mixed in the block; the magnitude of the frequency vectors represents the magnitude of the spatial frequency of the moiré pattern; the direction of the frequency vectors represents the direction of the spatial frequency of the moiré pattern.

[0068] Gabor filters are constructed based on the magnitude and direction of the extracted frequency vectors. Two different Gabor filters are constructed for the two sets of frequency vectors extracted in each block. Each block is filtered using the two constructed Gabor filters to obtain two sets of filtered fringes.

[0069] An iterative method based on relaxation labels is used to classify the two sets of filtered stripes obtained for each block. This separates the two sets of moiré patterns in the grayscale image of the moiré target region into two images, each containing only one set of moiré patterns. Specifically, the iterative method based on relaxation labels involves: pre-assigning initial labels (i.e., indicating the dimension to which the extracted frequency vectors belong) to each block; updating the labels based on the compatibility between the frequency vectors and a predefined compatibility matrix; constructing a compatibility matrix using the similarity between frequency vectors from adjacent blocks (i.e., the more similar the magnitude and direction of the two frequency vectors, the higher the compatibility value); adjusting the labels of the frequency vectors to better align with the labels of the frequency vectors from adjacent blocks and the compatibility matrix, thus achieving label consistency among all frequency vectors; and continuing the iteration process until the labels reach a stable state, allowing the frequency vectors of the two dimensions to be classified into different categories based on their similarity and consistency.

[0070] The generalized moiré feature representation module is used to provide feature representation for moiré patterns after feature separation at any perspective angle, including periodic contour functions and curvature functions, so as to effectively characterize moiré features at generalized perspective angles.

[0071] Specifically, the generalized moiré feature representation module provides feature representation for moiré patterns after feature separation at any perspective angle by using a periodic contour function p. m (x′) and the curvature function g m The moiré pattern m(x, y) is represented in a composite form as follows, where (x, y) represents the pixel coordinates of the image:

[0072] m(x, y) = p m (g m (x, y)).

[0073] Wherein, the periodic contour function p m The method for calculating (x′) is as follows:

[0074]

[0075] Wherein, the bending function g m The calculation process for (x, y) includes:

[0076] Perform a Hilbert transform on each row m(t) or column m(t) of each group of separated moiré patterns to obtain an analytic signal H(m)(t);

[0077] Extract the instantaneous angle of the analytical signal H(m)(t) to obtain the phase information of each row m(t) or column m(t);

[0078] The phase information of all rows or columns of the moiré pattern image is stitched together, and the resulting phase image is unwrapped to eliminate discontinuous 2π transitions in the phase image, resulting in a continuous phase information image.

[0079] Dividing the entire continuous phase information image by 2π yields the curvature function g. m (x, y).

[0080] The fine-grained feature point reconstruction module is used to perform super-resolution reconstruction of visual markers and extract sub-pixel level feature points. The sub-pixel level feature points of the visual markers reconstructed from each frame of the image are used as input to the PnP pose estimation algorithm to calculate the six-degree-of-freedom pose of the camera relative to the visual markers.

[0081] Specifically, the process of extracting sub-pixel level feature points in the fine-grained feature point reconstruction module includes:

[0082] According to the bending function g of the moiré pattern m The bending function g of (x, y) and camera color filter array (CFA) c Calculate the curvature function g of the visual marker projected onto the camera color filter array plane at (x, y). p (x, y), the calculation method is as follows:

[0083] g p (x, y) = g c (x, y)-g m (x, y)

[0084] Among them, the curvature function g of the two dimensions of the color filter array (horizontal stripes and vertical stripes) c (x, y) represent g respectively. c (x, y) = y and g c (x, y) = x;

[0085] The bending function g of visual markers p A new bending function g′ is obtained by bilinear upsampling (x, y). p (x, y), thereby improving the bending function from pixel-level resolution to sub-pixel-level resolution;

[0086] For the new bending function g′ p (x, y) performs a two-dimensional fitting operation, extending its range from the grayscale image region of the moiré target area to the original region of interest (ROI) where the moiré is located;

[0087] The expanded new bending function g′ p (x, y) and periodic contour function By performing composite processing, super-resolution visual labels can be obtained;

[0088] In the super-resolution visual marker image, the coordinates of the vertices closest to the four vertices of the original region of interest where the moiré pattern is located are selected as the sub-pixel level feature points (u) of the reconstructed visual marker. i v i (i = 1, 2, 3, 4) These sub-pixel-level feature points are used as input to the PnP pose estimation algorithm. The six-degree-of-freedom pose of the camera relative to the visual marker is calculated according to the following formula, including the rotation matrix R and the translation vector T:

[0089]

[0090] Where s is the scaling factor, K is the calibrated camera intrinsic parameter, (X i Y i Z i ) represents the prior 3D coordinates in the scene corresponding to the fine-grained feature points.

[0091] This invention also provides a pose estimation method based on moiré patterns applicable to generalized perspective angles, which, based on the above system, includes the following steps:

[0092] 1) Use a camera to capture video of visual markers in a pose detection scene based on camera-captured visual markers at a fixed frame rate. The visual marker regions in the video frames contain moiré patterns.

[0093] 2) Extract the original region of interest (ROI) where the moiré pattern is located for each frame of the image and further crop out the moiré pattern target region with a regular rectangular shape;

[0094] 3) Separate the features of the two dimensions of the mixed moiré pattern image in the target region;

[0095] 4) Calculate the bending function for the moiré pattern features of the two separated dimensions respectively;

[0096] 5) Calculate the bending function of the visual marker based on the bending function of the moiré pattern and the bending function of the camera color filter array, and combine it with the periodic contour function to reconstruct the sub-pixel level feature points of the visual marker.

[0097] 6) Based on the sub-pixel level feature points of the visual markers reconstructed from each frame of the image, a pose estimation algorithm is used to output the six-degree-of-freedom pose of the camera relative to the visual markers.

[0098] This invention has many specific applications. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of this invention, and these improvements should also be considered within the scope of protection of this invention.

Claims

1. A pose estimation system based on moiré patterns applicable to generalized perspective angles, characterized in that, In camera devices used for pose detection scenarios based on visual markers captured by cameras, the following modules are included: a moiré pattern hybrid feature separation module, a generalized moiré pattern feature representation module, and a fine-grained feature point reconstruction module. The moiré pattern hybrid feature separation module is used to extract and separate features from the hybrid moiré pattern image obtained when the camera captures visual markers; The generalized moiré feature representation module is used to provide feature representation for moiré patterns after feature separation at any perspective angle, including periodic contour functions and curvature functions, so as to effectively characterize moiré features at generalized perspective angles. The fine-grained feature point reconstruction module is used to perform super-resolution reconstruction of visual markers and extract sub-pixel level feature points. The sub-pixel level feature points of the visual markers reconstructed from each frame of the image are used as input to the PnP pose estimation algorithm to calculate the six-degree-of-freedom pose of the camera relative to the visual markers.

2. The pose estimation system based on moiré patterns applicable to generalized perspective angles according to claim 1, characterized in that, The visual marker is used to generate moiré patterns in the visual marker area of ​​the image when the camera captures the visual marker; the visual marker is a two-dimensional planar marker composed of one or more sets of periodic stripes with different directions, including a two-dimensional planar marker composed of two sets of periodic stripes with orthogonal directions; wherein each set of periodic stripes is composed of two linear stripes with high contrast colors, including black and white alternating linear stripes composed of black and white, and a pair of black and white stripes constitutes one cycle of the visual marker; the size and cycle of the visual marker are set according to the interaction distance requirements of different applications.

3. The pose estimation system based on moiré patterns applicable to generalized perspective angles according to claim 1, characterized in that, The hybrid moiré pattern image refers to a moiré pattern image in which two sets of periodic stripes overlap, wherein each set of periodic stripes is straight or curved.

4. The pose estimation system based on moiré patterns applicable to generalized perspective angles according to claim 1, characterized in that, The feature extraction in the moiré pattern hybrid feature separation module specifically includes: The quadrilateral detection algorithm is used to extract the original region of interest where the moiré pattern is located in each frame of the image, and the coordinates of the four vertices of the quadrilateral are recorded. A regular rectangle is extracted from the inside of the quadrilateral as the final moiré pattern target area. The color channel that makes the contrast of the moiré pattern stripes the strongest is selected to obtain a grayscale image of the moiré pattern target area. This grayscale image contains two mixed sets of moiré patterns.

5. The pose estimation system based on moiré patterns applicable to generalized perspective angles according to claim 4, characterized in that, The feature separation in the moiré pattern hybrid feature separation module specifically includes: The grayscale image of the moiré target region is divided into M×N blocks, and a 2D FFT operation is performed on each block to obtain the corresponding spectrogram. Extract two sets of frequency vectors from the spectrum, including the magnitude and direction of the frequency vectors; wherein, the two sets of frequency vectors represent the frequency characteristics of the two sets of moiré patterns mixed in the block; the magnitude of the frequency vectors represents the magnitude of the spatial frequency of the moiré pattern; the direction of the frequency vectors represents the direction of the spatial frequency of the moiré pattern. Gabor filters are constructed based on the magnitude and direction of the extracted frequency vectors. Two different Gabor filters are constructed for the two sets of frequency vectors extracted in each block. Each block is filtered using the two constructed Gabor filters to obtain two sets of filtered fringes. An iterative method based on relaxation labels is used to classify the two sets of stripes obtained after filtering for each block, and the two sets of moiré patterns contained in the grayscale image of the moiré pattern target region are separated into two images, each containing only one set of moiré patterns.

6. The pose estimation system based on moiré patterns applicable to generalized perspective angles according to claim 1, characterized in that, The generalized moiré feature representation module provides feature representation for moiré patterns after feature separation at any perspective angle, referring to the use of a periodic contour function p to represent moiré patterns with arbitrary curved shapes. m (x′) and the curvature function g m The moiré pattern m(x,y) is represented in a composite form as follows, where (x,y) represents the pixel coordinates of the image: m(x,y)=p m (g m (x,y))。 7. The pose estimation system based on moiré patterns applicable to generalized perspective angles according to claim 6, characterized in that, The periodic contour function p m The method for calculating (x′) is as follows:

8. The pose estimation system based on moiré patterns applicable to generalized perspective angles according to claim 6, characterized in that, The bending function g m The calculation process for (x,y) includes: Perform a Hilbert transform on each row m(t) or column m(t) of each group of separated moiré patterns to obtain an analytic signal H(m)(t); Extract the instantaneous angle of the analytical signal H(m)(t) to obtain the phase information of each row m(t) or column m(t); The phase information of all rows or columns of the moiré pattern image is stitched together, and the resulting phase image is unwrapped to eliminate discontinuous 2π transitions in the phase image, resulting in a continuous phase information image. Dividing the entire continuous phase information image by 2π yields the curvature function g. m (x,y).

9. The pose estimation system based on moiré patterns applicable to generalized perspective angles according to claim 6, characterized in that, The process of extracting sub-pixel level feature points in the fine-grained feature point reconstruction module includes: According to the bending function g of the moiré pattern m The curvature function g of (x,y) and camera color filter array (CFA) c Calculate the curvature function g of the visual marker projected onto the camera color filter array plane (x, y). p (x, y), the calculation method is as follows: g p (x,y)=g c (x,y)-g m (x,y) Among them, the curvature function g of the two-dimensional stripes of the color filter array c (x, y) represent g respectively. c (x,y) = y and g c (x,y)=x; The bending function g of visual markers p A new bending function g′ is obtained by bilinear upsampling (x,y). p (x,y), thereby improving the bending function from pixel-level resolution to sub-pixel-level resolution; For the new bending function g′ p (x,y) performs a two-dimensional fitting operation, extending its range from the grayscale image region of the moiré target region to the original region of interest where the moiré is located; The expanded new bending function g′ p (x,y) and periodic contour function By performing composite processing, super-resolution visual labels can be obtained; In the super-resolution visual marker image, the coordinates of the vertex closest to the four vertices of the original region of interest where the moiré pattern is located are selected as the sub-pixel level feature points of the reconstructed visual marker. These sub-pixel level feature points are used as the input of the PnP pose estimation algorithm to obtain the six-degree-of-freedom pose of the camera relative to the visual marker.

10. A pose estimation method based on moiré patterns applicable to generalized perspective angles, based on the system described in any one of claims 1-9, characterized in that, The steps include the following: 1) Use a camera to capture video of visual markers in a pose detection scene based on camera-captured visual markers at a fixed frame rate. The visual marker regions in the video frames contain moiré patterns. 2) Extract the original region of interest where the moiré pattern is located for each frame of the image and further crop out the target region of the moiré pattern with a regular rectangular shape; 3) Separate the features of the two dimensions of the mixed moiré pattern image in the target region; 4) Calculate the bending function for the moiré pattern features of the two separated dimensions respectively; 5) Calculate the bending function of the visual marker based on the bending function of the moiré pattern and the bending function of the camera color filter array, and combine it with the periodic contour function to reconstruct the sub-pixel level feature points of the visual marker. 6) Based on the sub-pixel level feature points of the visual markers reconstructed from each frame of the image, a pose estimation algorithm is used to output the six-degree-of-freedom pose of the camera relative to the visual markers.

Images