A Moire-based Pose Estimation System and Method

Abstract

The present invention discloses a moiré-based pose estimation system and method, including: an image preprocessing module for processing an image with a screen as the main body captured by a camera to extract an effective central region and moiré features; a position estimation module for selecting interest points on the preprocessed image, calculating the estimated distance from the camera to the points on the screen corresponding to each interest point, and finally calculating the relative position between the camera and the screen according to the multilateration method; and an attitude estimation module for calculating and estimating the relative attitude between the camera and the screen based on the camera optical axis and roll angle measurement. The present invention utilizes the moiré formed by the superposition of the color filter array at the front end of the camera and the screen projection pixels, and can realize the six-degree-of-freedom relative pose estimation between the camera and the screen without relying on visual markers.

Description

Technical Field

[0001] The present invention belongs to the technical field of position and posture estimation, and in particular relates to a system and method for estimating the relative position and posture of a screen and a camera based on moiré patterns. Background Art

[0002] Today, six-degree-of-freedom pose estimation has become a key technology in fields such as human-computer interaction (HCI) and mobile robot positioning. Among them, a typical and widespread application scenario is somatosensory games and augmented reality (AR) applications based on screen interaction: the user's head-mounted device and handles are usually embedded with cameras, and the laser base station uses multiple high-speed light scans to accurately reconstruct the six-degree-of-freedom (6-DoF) posture of the camera to provide an immersive experience. For some routine interactive actions, the existing pose estimation accuracy can basically meet the application requirements. However, for some precise operations such as aiming, shooting, steering and other interactive actions with high precision requirements, the current camera tracking technology is still needed to continuously improve the accuracy of the six-degree-of-freedom pose detection algorithm.

[0003] Existing 6-DoF pose detection technologies based on wearable devices mainly include inertial measurement unit (IMU)-based and vision-based perception methods:

[0004] An IMU is an inertial sensor that uses an accelerometer to detect linear acceleration and a gyroscope to detect angular velocity. Position and angle must be restored through integration. However, due to interference from factors such as temperature, zero bias, and vibration, the calculated translation and rotation often suffer from integral drift. Even small measurement errors can accumulate into significant deviations over time. Furthermore, the nature of IMU inertial measurement makes accelerometers and gyroscopes less sensitive to small and slow motion measurements.

[0005] Existing vision-based camera 6-DoF pose estimation techniques mainly include methods based on Perspective-n-Point (PnP) and deep learning-based methods. The PnP-based pose estimation method estimates the six-degree-of-freedom pose of the camera by finding the scaling and deformation relationship of feature points from the 3D world to the 2D image. However, this solution relies on prior knowledge such as the positions and layouts of marker points in the 3D world. Once the detection environment changes, both the deployment and measurement processes must be redone. Deep learning-based methods use trained neural network models to predict the camera pose corresponding to a given image, but there are problems such as high model training complexity and weak model generalization ability. In addition, recognizing tiny pose changes is a challenge for vision-based solutions, especially when the camera is in front of the marker points and at a relatively far distance. Whether it is a vision-based localization method based on PnP or machine learning, its essence is to perform spatial domain analysis on images, so the accuracy is limited to the pixel level. Such accuracy can meet general applications such as human-computer interaction, but it is far from sufficient to achieve six-degree-of-freedom pose detection for precise and tiny movements. Summary of the Invention

[0006] Aiming at the deficiencies of the above-mentioned existing technologies, the purpose of the present invention is to provide a moiré-based pose estimation system and method to solve problems such as insufficient accuracy, high cost, and poor robustness of existing pose estimation methods.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A moiré-based pose estimation system of the present invention is applied to a camera device in a camera and screen interaction scenario, and includes: an image preprocessing module, a position estimation module, and a pose estimation module;

[0009] The image preprocessing module is used to preprocess the image with the screen as the main body captured by the camera; specifically: extract the effective image central region and determine its boundary, perform image equalization, image enhancement, and frequency domain analysis on the image central region, and extract the moiré features of the image central region: the spatial frequency f m and the frequency propagation direction θ m ;

[0010] The position estimation module is used to estimate the three-degree-of-freedom relative position between the camera and the screen; specifically: select interest points on the boundary of the image central region and divide interest regions centered on the interest points; perform single-point ranging on all interest regions respectively, and obtain the estimated distance d between the point S i on the screen corresponding to the center of each interest region and the camera i, all the estimated distances together constitute an estimated distance array; based on the obtained estimated distance array and the multilateration method, the position (x, y, z) of the camera in the screen coordinate system Oxyz is estimated;

[0011] An attitude estimation module, used to estimate the three - degree - of - freedom relative attitude between the camera and the screen; specifically: based on the position (x, y, z) of the camera in the screen coordinate system Oxyz, the optical axis direction of the camera is calculated, and then combined with the roll angle measurement, the vector representation [u, v, w] of the three - axis azimuth vector corresponding to the camera attitude in the screen coordinate system Oxyz is derived.

[0012] Further, the origin O in the screen coordinate system Oxyz is the intersection point of the optical axis and the screen plane when the camera captures an image, that is, the projection of point O on the image is located at the mid - point of the image; the X - axis and Y - axis of the screen coordinate system Oxyz are parallel to the long side and short side of the screen respectively, and the Z - axis is orthogonal to the XOY plane.

[0013] Further, the camera device includes but is not limited to a camera module with image acquisition and data transmission functions and mobile devices with a camera module, such as smartphones, smart watches, smart glasses, head - mounted display devices, smart handles, mobile robots, intelligent robotic arm devices, etc.

[0014] Further, the screen refers to a display screen with regular pixel arrangements other than the silver screen, and mobile devices with a display screen module, such as smartphones, smart watches, tablets, etc.; differences in the display technology, pixel geometry, and pixel arrangement of the screen device are allowed; the display technology covers a variety of screen display technologies in major categories such as LCD, OLED, and LED technologies.

[0015] Further, the pre - processing process of the image pre - processing module is specifically as follows:

[0016] Extract the central area of the image captured by the camera with the screen as the main body and record the boundary of the central area to eliminate the distortion and vignetting around the image caused by the lens lens effect; the number of pixels P of the side length of the central area c and the number of pixels P of the short side of the image m The ratio is denoted as σ;

[0017] Perform histogram equalization on the extracted central area to enhance the contrast;

[0018] Perform image enhancement on the equalized image, including adaptive binarization and median filtering;

[0019] Perform a fast Fourier transform on the enhanced image to obtain a spectrogram containing moiré patterns;

[0020] Adaptive binarization is performed on the spectrogram to extract the pulse region with the strongest energy; the moiré characteristics are calculated based on the central coordinates (u, v) of the pulse region: the spatial frequency f m and the frequency propagation direction θ m , and the specific calculation method is as follows:

[0021]

[0022]

[0023] Furthermore, the method for selecting the interest points and dividing the interest regions in the position estimation module is as follows:

[0024] Points located on the boundary of the central region are selected dispersedly, that is, points I U 、I D 、I L 、I R , which together form an interest point array;

[0025] Centered on each interest point, a temporary interest region is divided for each interest point with the side length pixel number P c of the central region as the side length; and the area of the temporary interest region is reduced;

[0026] The Fourier transform is performed on the temporary interest region, the central coordinates (u′, v′) of the pulse region with the strongest energy in the spectrogram are extracted, and the spatial frequency of the temporary interest region is calculated

[0027] The side length pixel number R′ of the temporary interest region is set to 4f′ to obtain the final interest region.

[0028] Furthermore, the single-point ranging method in the position estimation module is as follows:

[0029] For the interest points I U 、I D 、I L 、I R corresponding interest regions R U 、R D 、R L 、R R , a fast Fourier transform is performed to obtain a spectrogram;

[0030] Adaptive binarization is performed on the spectrogram to extract two pulse regions C h and C v with the strongest energy, and the included angle formed by the lines connecting the two pulse regions with the strongest energy to the origin is 90 degrees; the pulse region C h reflects the moiré characteristics of the horizontal moiré stripes, and the pulse region C v reflects the moiré characteristics of the vertical moiré stripes;

[0031] Based on the central coordinates (u h , v h ) and (u v , v v ) of the two pulse regions with the strongest energies, a pair of moiré features <f, θ> are calculated respectively, including the spatial frequency f and the propagation direction θ:

[0032]

[0033]

[0034]

[0035]

[0036] In the formula, <f h , θ h > represents a pair of moiré features calculated from the coordinates (u h , v h ); <f v , θ v > represents a pair of moiré features calculated from the coordinates (u v , v v ).

[0037] Based on the spatial frequency f and the propagation direction θ of the two pulse regions with the strongest energies, the distance from the point on the screen corresponding to the center (i.e., the point of interest) of the region of interest to the camera is further calculated:

[0038]

[0039]

[0040] In the formula, d h represents the distance calculated from the pulse region C h ; d v represents the distance calculated from the pulse region C v .

[0041] For the points of interest I L and I R , d h is selected as the estimated distance from the point on the screen corresponding to the point of interest to the camera; for the points of interest I V and I D , d v is selected as the estimated distance from the point on the screen corresponding to the point of interest to the camera.

[0042] Furthermore, the multi - lateral positioning method in the position estimation module is specifically as follows:

[0043] Define the coordinates of the position P of the camera in the screen coordinate system Oxyz as [x, y, z] T ;

[0044] Define the point S on the screen corresponding to each point of interest i with coordinates [x i , y i , z i T ; Calculate using the points S on the screen corresponding to the points of interest U (0, a, 0), S D (0, -a, 0), S L (-a, 0, 0), S R (a, 0, 0), where the parameter where σ represents the number of pixels of the side length of the central region P c compared with the number of pixels of the shorter side of the image P m ratio; L c represents the size of the color filter array of the camera, f represents the physical focal length of the camera, d0 represents the distance from the camera to the origin O of the screen coordinates, and is calculated according to the extracted moiré pattern features f m and θ m as follows:

[0045]

[0046] The theoretical distance from the position P of the camera to each point S i is expressed as ||P - S i ||2, and all the theoretical distances together form a theoretical distance array;

[0047] Denote the estimated distance between the camera position P and the point S on the screen corresponding to the point of interest as d i , and the camera position corresponding to the minimum difference between the theoretical distance array and the estimated distance array is the optimal camera estimated position; the optimal solution P i is calculated by the following formula: * as follows:

[0048]

[0049] Furthermore, the method for the attitude estimation module to estimate the three-degree-of-freedom attitude of the camera relative to the screen is:

[0050] Calculate the vector according to the position P of the camera in the screen coordinate system obtained by the position estimation module Define the optical axis of the camera w = [w x , w y , w z T ​​, the unit vector of the optical axis w

[0051] Calculate the roll angle θ of the camera rotating around the optical axis w c , that is, the frequency propagation direction θ of the camera color filter array c , and the calculation formula is as follows:

[0052]

[0053] Among them, θ m represents the spatial frequency propagation direction of the moiré pattern; f s represents the spatial frequency of the screen pixel array, f c represents the spatial frequency of the camera color filter array; f represents the physical focal length of the camera; d0 represents the distance from the camera to the origin O of the screen coordinate system. According to the spatial frequency f m of the moiré pattern feature and the frequency propagation direction θ m it is calculated as:

[0054]

[0055] Define the projection of the X-axis of the screen coordinate system on the plane of the camera color filter array Since is the projection of the X-axis, so M y = 0; and because is orthogonal to the optical axis w = [w x , w y , w z T , then Further, the unit vector of can be calculated

[0056] According to the Rodrigues rotation formula, the unit vector of the azimuth axis u of the camera attitude is obtained by rotating the unit vector around the unit vector by the roll angle θ c as follows:

[0057]

[0058] The last azimuth axis v-axis of the camera attitude is obtained by the cross product of the optical axis w and the azimuth axis u.

[0059] ​The pose estimation in this invention refers to estimating the relative position and relative pose between the camera and the screen. It is applicable to scenarios where the screen is fixed and the camera moves, as well as scenarios where the camera is fixed and the screen moves. The relative pose relationship between the camera and the screen in this invention is characterized by the three-degree-of-freedom position (x, y, z) and the three-degree-of-freedom pose [u, v, w] of the camera in the screen coordinate system Oxyz; for each screen image with moiré patterns, a moiré-based pose estimation system proposed in this invention can output the six-degree-of-freedom pose of the camera relative to the O point of the screen (whose projection is at the center of the image), including the three-degree-of-freedom position and the three-degree-of-freedom pose; since the output six-degree-of-freedom pose represents a relative relationship, it is also easy to derive the six-degree-of-freedom pose of the screen in the camera coordinate system.

[0060] This invention also provides a moiré-based pose estimation method. Based on the above system, it includes the following steps:

[0061] 1) Use the camera to collect videos of the interacting screen at a fixed frame rate. The video frames contain moiré images with low spatial frequencies.

[0062] 2) Perform image preprocessing on each frame of the video frame sequence, and execute interest point and interest region extraction on the processed image, that is, select interest points and divide interest regions.

[0063] 3) Perform single-point ranging on the interest region corresponding to each interest point. Extract moiré feature parameters through frequency domain analysis, and then calculate the estimated distance from the camera to the point on the screen corresponding to each interest point to obtain an estimated distance array.

[0064] 4) Based on the estimated distance array, use the multilateration method to estimate the three-degree-of-freedom relative position between the camera and the screen.

[0065] 5) Calculate the roll angle of the camera according to the moiré features. Based on the position and roll angle of the camera, estimate the three-degree-of-freedom relative pose between the camera and the screen.

[0066] The beneficial effects of this invention:

[0067] 1. Ultra-high precision: The moiré patterns formed by the superposition of the color filter array at the front end of the camera and the screen projection pixels are extremely sensitive to the relative pose changes between the camera and the screen; any slight change in the relative position and pose will cause significant changes in the moiré features.

[0068] 2. Robustness: Compared with traditional feature-based computer vision methods, the statistical features of moiré stripes in the frequency domain can provide more accurate and robust information for pose estimation, while the number of feature points in traditional computer vision methods is limited and is easily affected by the environment.

[0069] 3. Low cost: No additional signal transmitting and receiving equipment is required to achieve real-time six-degree-of-freedom posture detection of the camera and interactive screen.

[0070] 4. Marker-free: Unlike traditional positioning methods based on visual markers, the present invention does not require prior calibration of feature points in three-dimensional space and has scalability across equipment and environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 2 is a diagram illustrating the structure of a pose estimation system according to an embodiment of the present invention;

[0072] Figure 2 This is a schematic diagram of the image preprocessing principle of the pose estimation system of the present invention;

[0073] Figure 3 Schematic diagram of interest point and interest area extraction in the pose estimation system of the present invention;

[0074] Figure 4 This is a flow chart of the single-point ranging method of the posture estimation system of the present invention;

[0075] Figure 5 Schematic diagram of the single-point ranging method of the pose estimation system of the present invention

[0076] Figure 6 This is a schematic diagram of the multi-lateral positioning method of the pose estimation system of the present invention;

[0077] Figure 7 This is a flow chart of the pose estimation system of the present invention;

[0078] Figure 8 This is a diagram showing the pose estimation principle of the pose estimation system of the present invention. DETAILED DESCRIPTION

[0079] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0080] The relative positional relationship between the camera and the screen is characterized as the camera's three-degree-of-freedom position (x, y, z) and three-degree-of-freedom attitude [u, v, w] in the screen coordinate system Oxyz. The origin O in the screen coordinate system Oxyz is the intersection of the optical axis and the screen plane when the camera captures the image, meaning that the projection of point O onto the image is located at the image's midpoint. For each screen image with moiré patterns, the present invention can output the camera's six-degree-of-freedom pose relative to point O on the screen (its projection is at the image center), including the three-degree-of-freedom position and three-degree-of-freedom attitude. Because the output six-degree-of-freedom pose represents a relative relationship, the six-degree-of-freedom pose of the screen relative to the camera coordinate system can also be easily derived.

[0081] ReferenceFigure 1 As shown in Figure 1 , a moiré-based pose estimation system of the present invention is applied to a camera device in a camera-screen interaction scenario, including: an image preprocessing module, a position estimation module, and an attitude estimation module;

[0082] Among them, the camera device includes but is not limited to a camera module with image acquisition and data transmission functions and mobile devices with a camera module, such as smartphones, smart watches, smart glasses, head-mounted display devices, smart handles, mobile robots, intelligent robotic arm devices, etc.

[0083] The screen refers to a display screen with regular pixel arrangements except for the silver screen, as well as mobile devices with a display screen module, such as smartphones, smart watches, and tablets. Differences in the display technology, pixel geometry, and pixel arrangement of the screen device are allowed. The display technology covers a variety of screen display technologies in the large categories of LCD, OLED, LED, etc.

[0084] The image preprocessing module is used to preprocess the image with the screen as the main body captured by the camera, extract the effective image center region and determine its boundary; specifically: perform image equalization, image enhancement, and frequency domain analysis on the image center region, and extract the moiré features of the image center region: spatial frequency f m and frequency propagation direction θ m ;

[0085] The position estimation module is used to estimate the three-degree-of-freedom relative position between the camera and the screen; specifically: select points of interest (POI) on the boundary of the image center region and divide the region of interest (ROI) with the points of interest as the center; perform single-point ranging on all regions of interest respectively, and obtain the estimated distance d between the point S on the screen corresponding to the center of each region of interest (i.e., POI) and the camera i All the estimated distances together form an estimated distance array; based on the obtained estimated distance array and the multilateration method, estimate the position (x, y, z) of the camera in the screen coordinate system Oxyz; i

[0086] The attitude estimation module is used to estimate the three-degree-of-freedom relative attitude between the camera and the screen; specifically: calculate the optical axis direction of the camera based on the position (x, y, z) of the camera in the screen coordinate system Oxyz, and then combine the roll angle measurement to derive the vector representation [u, v, w] of the three-axis azimuth vector corresponding to the camera attitude in the screen coordinate system Oxyz.

[0087] Refer to Figure 2 As shown, the preprocessing process of the image preprocessing module is specifically as follows:

[0088] Extract the central region from the image captured by the camera with the screen as the main body and record the boundary of the central region to eliminate the distortion and vignetting around the image caused by the lens effect; the number of pixels P on the side length of the central region c and the number of pixels P on the short side of the image m The ratio is denoted as σ;

[0089] Perform histogram equalization on the extracted central region to enhance the contrast;

[0090] Perform image enhancement on the equalized image, including adaptive binarization and median filtering. Based on the image enhancement operation, the original content displayed on the screen except for moiré can be effectively eliminated;

[0091] Perform a fast Fourier transform on the enhanced image to obtain a spectrogram containing moiré features;

[0092] Perform adaptive binarization on the spectrogram to extract the pulse region with the strongest energy; calculate the moiré features: spatial frequency f m and the frequency propagation direction θ m , the specific calculation method is:

[0093]

[0094]

[0095] Refer to Figure 3 As shown, the method for selecting interest points and dividing interest regions in the position estimation module is:

[0096] Dispersedly select points located on the boundary of the central region, that is, points I U , I D , I L , I R , which together form an interest point array;

[0097] Centered on each interest point, divide a temporary interest region for each interest point with the side length pixel number Pc of the central region; and reduce the area of the temporary interest region;

[0098] Perform a Fourier transform on the temporary interest region, extract the central coordinates (u′, v′) of the pulse region with the strongest energy in the spectrogram, and calculate the spatial frequency of the temporary interest region

[0099] Set the side length pixel number R′ of the temporary interest region to 4f′ to obtain the final interest region.

[0100] Refer to Figure 4 As shown, the single-point ranging method in the position estimation module is:

[0101] For the point of interest I U 、I D 、I L 、I R corresponding region of interest R U 、R D 、R L 、R R perform a fast Fourier transform to obtain a spectrogram; take the region of interest R R where the point of interest I R is located as an example for display, and the schematic diagram is as shown in Figure 5 ;

[0102] Perform adaptive binarization on the spectrogram to extract two pulse regions C h and C v with the strongest energy, and the angle formed by the lines connecting the two pulse regions with the strongest energy to the origin is 90 degrees; the pulse region C h reflects the moiré characteristics of the horizontal moiré pattern strip, and the pulse region C v reflects the moiré characteristics of the vertical moiré pattern strip;

[0103] According to the central coordinates (u h , v h ) and (u v , v v ) of the two pulse regions with the strongest energy, calculate a pair of moiré characteristics <f, θ> respectively, including the spatial frequency f and the propagation direction θ:

[0104]

[0105]

[0106]

[0107]

[0108] In the formula, <f h , θ h > represents a pair of moiré characteristics calculated from the coordinates (u h , v h ); <f v , θ v > represents a pair of moiré characteristics calculated from the coordinates (u v , v v );

[0109] According to the spatial frequency f and the propagation direction θ of the two pulse regions with the strongest energy, further calculate the distance from the point on the screen corresponding to the center (i.e., the point of interest) of the region of interest to the camera:

[0110]

[0111]

[0112] In the formula, d h represents the distance calculated from the pulse region C h ; d v represents the distance calculated from the pulse region C v ;

[0113] For the points of interest I L and I R , select d h as the estimated distance from the point on the screen corresponding to the point of interest to the camera; for the points of interest I U and I D , select d v as the estimated distance from the point on the screen corresponding to the point of interest to the camera.

[0114] Refer to Figure 6 as shown, the multi-lateral positioning method in the position estimation module is specifically as follows:

[0115] Define the coordinates of the position P of the camera in the screen coordinate system Oxyz as [x, y, z] T ;

[0116] Define the coordinates of the point S i on the screen corresponding to each point of interest as [x i , y i , z i T ; Calculate with the points S U (0, a, 0), S D (0, -a, 0), S L (-a, 0, 0), S R (a, 0, 0) on the screen corresponding to the points of interest, and the parameter where σ represents the ratio of the side length pixel number P c of the central region to the short side pixel number P m of the image; L c represents the size of the color filter array of the camera, f represents the physical focal length of the camera, d0 represents the distance from the camera to the origin O of the screen coordinates, and is calculated according to the extracted moiré pattern features f m and θ m :

[0117]

[0118] The theoretical distance from the position P of the camera to each point S i is expressed as ||P - S​i ||2. All the theoretical distances together form a theoretical distance array;

[0119] Denote the estimated distance between the camera position P and the point S on the screen corresponding to the point of interest as d i ; when the difference between the theoretical distance array and the estimated distance array is the smallest, the corresponding camera position is the optimal camera estimation position; the optimal solution P i is calculated by the following formula: * Refer to

[0120]

[0121] As shown in Figure 7 and Figure 8 , the method for the pose estimation module to estimate the three-degree-of-freedom pose of the camera relative to the screen is as follows:

[0122] Calculate the vector according to the position P of the camera in the screen coordinate system calculated in the position estimation module. Define the optical axis of the camera w = [w x , w y , w z T , then the unit vector

[0123] Calculate the roll angle θ of the camera rotating around the optical axis w c , that is, the frequency propagation direction θ of the color filter array of the camera c , and the calculation formula is as follows:

[0124]

[0125] Among them, θ m represents the spatial frequency propagation direction of the moiré pattern; f s represents the spatial frequency of the screen pixel array, f c represents the spatial frequency of the color filter array of the camera; f represents the physical focal length of the camera; d0 represents the distance from the camera to the origin O of the screen coordinate system. According to the spatial frequency f m of the moiré pattern feature and the frequency propagation direction θ m , it is calculated as:

[0126]

[0127] Define the projection of the X-axis of the screen coordinate system on the plane of the color filter array of the camera Since is the projection of the X-axis, so M y = 0; and because is orthogonal to the optical axis then Furthermore, it can be calculated that Unit vector

[0128] According to Rodrigues' rotation formula, the unit vector of the azimuth axis u of the camera pose From the unit vector Rotate around the unit vector [[ID=ll]]by the roll angle θ c to obtain:

[0129]

[0130] The last azimuth axis v-axis of the camera pose is obtained by the cross product of the optical axis w and the azimuth axis u

[0131] The present invention also provides a moiré-based pose estimation method. Based on the above system, it includes the following steps:

[0132] 1) Use the camera to perform video acquisition on the interactive screen at a fixed frame rate. The video frames contain moiré images with low spatial frequencies;

[0133] 2) Perform image preprocessing on each frame of the video frame sequence, and perform interest point and interest area extraction on the processed image, that is, select interest points and divide interest areas;

[0134] 3) Perform single-point ranging on the interest area corresponding to each interest point, extract moiré feature parameters through frequency domain analysis, and then calculate the estimated distance from the camera to the point on the screen corresponding to each interest point to obtain an estimated distance array;

[0135] 4) Based on the estimated distance array, use the multilateration method to estimate the three-degree-of-freedom relative position between the camera and the screen;

[0136] 5) Calculate the roll angle of the camera according to the moiré features, and estimate the three-degree-of-freedom relative pose between the camera and the screen based on the position and roll angle of the camera

[0137] The specific application ways of the present invention are numerous. The above description is only the preferred implementation manner of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements can be made, and these improvements should also be regarded as the protection scope of the present invention

Claims

1. A moiré-based pose estimation system, applied to a camera device in a camera-screen interaction scenario, characterized in that: include: Image preprocessing module, position estimation module and posture estimation module; An image preprocessing module is used to preprocess the image with the screen as the main body captured by the camera; Specifically, the effective image center area is extracted and its boundary is determined. Image equalization, image enhancement, and frequency domain analysis are performed on the image center area to extract the moiré features of the image center area: spatial frequency f m and the frequency propagation direction θ m ; The position estimation module is used to estimate the three-degree-of-freedom relative position between the camera and the screen; specifically, the module selects a point of interest on the boundary of the central area of the image and divides the region of interest as the center of the point of interest; performs single-point ranging on all regions of interest, and obtains the point S on the screen corresponding to the center of each region of interest. i Estimated distance d to the camera i , all estimated distances together constitute the estimated distance array; Estimate the camera's position (x, y, z) in the screen coordinate system Oxyz based on the estimated distance array and multilateration. The posture estimation module is used to estimate the three-degree-of-freedom relative posture between the camera and the screen. Specifically, the optical axis direction of the camera is calculated based on the position (x, y, z) of the camera in the screen coordinate system Oxyz, and the vector representation [u, v, w] of the three-axis orientation vector corresponding to the camera posture in the screen coordinate system Oxyz is derived in combination with the roll angle measurement.

2. The moiré-based pose estimation system according to claim 1, wherein: The origin O in the screen coordinate system Oxyz is the intersection of the optical axis and the screen plane when the camera captures the image, that is, the projection of point O on the image is located at the midpoint of the image; the X axis and X axis of the screen coordinate system Oxyz are parallel to the long side and short side of the screen respectively, and the Z axis is orthogonal to the XOY plane.

3. The moiré-based pose estimation system according to claim 1, wherein: The preprocessing process of the image preprocessing module is specifically as follows: The center area of the image captured by the camera with the screen as the main body is extracted and the boundary of the center area is recorded to eliminate the image distortion and dark corners caused by the lens effect; the side length of the center area is the number of pixels P c The number of pixels on the shorter side of the image P m The ratio is denoted as σ; Perform histogram equalization on the extracted central area to enhance the contrast; Perform image enhancement on the equalized image, including adaptive binarization and median filtering; Perform fast Fourier transform on the enhanced image to obtain a spectrum containing moiré features; Adaptively binarize the spectrum to extract the pulse area with the strongest energy; Calculate the moiré feature based on the center coordinates (u, v) of the pulse area: spatial frequency f m and the frequency propagation direction θ m , the specific calculation method is:

4. The moiré-based pose estimation system according to claim 3, wherein: The method for selecting points of interest and dividing regions of interest in the position estimation module is: Dispersively select points located on the boundary of the central area, i.e., point I U , I D , I L , I R , together forming an array of points of interest; With each point of interest as the center, the number of pixels P of the side length of the central area c Divide each point of interest into a temporary region of interest based on the side length; and reduce the area of the temporary region of interest; Perform Fourier transform on the temporary region of interest, extract the center coordinates (u′, v′) of the pulse region with the strongest energy in the spectrum graph, and calculate the spatial frequency of the temporary region of interest The number of pixels of the side length R′ of the temporary region of interest is set to 4f′ to obtain the final region of interest.

5. The moiré-based pose estimation system according to claim 4, wherein: The single point ranging method in the position estimation module is: Points of Interest I U , I D , I L , I R The corresponding region of interest R U 、R D 、R L 、R R Perform fast Fourier transform to obtain the spectrum diagram; Adaptively binarize the spectrum and extract the two pulse regions C with the strongest energy h and C v , and the angle between the line connecting the two pulse regions with the strongest energy and the origin is 90 degrees; pulse region C h Moiré characteristics of the horizontal moiré spline, pulse area C v Moiré features reflecting vertical moiré splines; According to the center coordinates of the two most energetic pulse regions (u h , v h ) and (u v , v v ), calculate a pair of moiré features respectively<f,θ> , including the spatial frequency f and the propagation direction θ: Where, <f h ,θ h > represents the coordinates (u h , v h ) A pair of moiré features calculated; <f v ,θ v > represents the coordinates (u v , v v ) A pair of moiré features calculated; Based on the spatial frequency f and propagation direction θ of the two pulse regions with the strongest energy, the distance from the point on the screen corresponding to the center of the area of interest to the camera is further calculated: Where, d h Indicated by the pulse area c h The calculated distance; d v Indicated by the pulse area C v The calculated distance; For points of interest I L and I R , select d h As the estimated distance between the point on the screen corresponding to the point of interest and the camera; U and I D , select d v The estimated distance from the point on the screen corresponding to the point of interest to the camera.

6. The moiré-based pose estimation system according to claim 5, characterized in that: The multilateration positioning method in the position estimation module is specifically as follows: Define the camera position P in the screen coordinate system Oxyz as [x, y, z] T ; Define the point S on the screen corresponding to each point of interest i The coordinates of [x i ,y i , z i ] T ; Point S on the screen corresponding to the point of interest U (0, a, 0), S D (0, -a, 0), S L (-a, 0, 0), S R (a, 0, 0) is used for calculation, and the parameters Where σ represents the number of pixels of the side length of the central area P c The number of pixels on the shorter side of the image P m Ratio; L c Indicates the size of the camera color filter array, f indicates the physical focal length of the camera, d0 indicates the distance from the camera to the screen coordinate origin O, and according to the extracted moiré feature f m and θ m Calculation yields: From the camera position P to each point S i The theoretical distance is expressed as ||PS i ||2, all theoretical distances together constitute the theoretical distance array; Move the camera position P to the point S on the screen corresponding to the point of interest i The estimated distance between i , when the difference between the theoretical distance array and the estimated distance array is the smallest, the corresponding camera position is the optimal camera estimated position; the optimal solution P * Calculated by the following formula:

7. The moiré-based pose estimation system according to claim 6, wherein: The method by which the posture estimation module estimates the three-degree-of-freedom posture of the camera relative to the screen is: Calculate the vector based on the position P of the camera in the screen coordinate system calculated in the position estimation module Define the optical axis of the camera w=[w x , w y , w z ] T , then the unit vector of the optical axis w is Calculate the roll angle θ of the camera around the optical axis w c , that is, the frequency propagation direction θ of the camera color filter array c , the calculation formula is as follows: Among them, θ m Indicates the spatial frequency propagation direction of the moiré pattern; f s Represents the spatial frequency of the screen pixel array, f c Represents the spatial frequency of the camera's color filter array; f represents the physical focal length of the camera; d0 represents the distance from the camera to the origin O of the screen coordinate system. According to the spatial frequency f of the moiré feature m and the frequency propagation direction θ m Calculation yields: Define the projection of the X-axis of the screen coordinate system on the camera color filter array plane because is the projection of the X axis, so M y =0; and because Orthogonal to the optical axis w=[w x , w y , w z ] T ,but Further calculation The unit vector According to the Rodrigues rotation formula, the unit vector of the azimuth axis u of the camera posture By unit vector Around a unit vector Roll angle θ c get: The final orientation axis v of the camera pose is obtained by the cross product of the optical axis w and the orientation axis u.

8. A method for pose estimation based on moiré patterns, based on the system according to any one of claims 1 to 7, characterized in that: The steps are as follows: 1) Using a camera to capture video of the interactive screen at a fixed frame rate, the video frame contains a moiré pattern image with a low spatial frequency; 2) Perform image preprocessing on each frame of the video frame sequence, and perform interest point and interest region extraction on the processed image, that is, selecting interest points and dividing interest regions; 3) Perform single-point ranging on the area of interest corresponding to each point of interest, extract moiré feature parameters through frequency domain analysis, and then calculate the estimated distance from the camera to the point on the screen corresponding to each point of interest to obtain an estimated distance array; 4) Based on the estimated distance array, the three-degree-of-freedom relative position between the camera and the screen is estimated using multilateration. 5) Calculate the roll angle of the camera based on the moiré pattern features, and estimate the three-degree-of-freedom relative posture between the camera and the screen based on the camera position and roll angle.

Images