High-efficiency somatosensory interaction prop and physical space calibration method thereof

Abstract

This invention discloses an efficient motion-sensing interactive prop and its physical space calibration method. The calibration method includes: S1. acquiring the intrinsic parameters of the camera, the extrinsic parameters between cameras, and the extrinsic parameters between the camera and the display; S2. collecting data on the movement of the interactive prop; S3. calculating the three-dimensional coordinate position of the light-emitting point using the camera's intrinsic parameters and the extrinsic parameters between the camera and the display, and transforming the three-dimensional coordinate position of the light-emitting point to a spatial position with the display center as the center of the world coordinate system, thus obtaining the spatial position of the light-emitting point in the display's world coordinate system; S4. interpolating the attitude data of the cascaded inertial sensor (IMU) to the three-dimensional coordinate data of the light-emitting point according to a second timestamp, thus obtaining the attitude data after interpolation by the cascaded inertial sensor (IMU); S5. constructing a system of nonlinear equations and solving the system of nonlinear equations to obtain the initial position, initial attitude of the interactive prop, and the relative coordinates of the light-emitting point in space.

Description

Technical Field

[0001] This invention relates to the field of virtual-real interaction technology, and in particular to an efficient motion-sensing interactive prop and its physical space calibration method. Background Technology

[0002] Motion-sensing interactive games achieve in-game interaction by capturing the player's body movements or using specially designed game props. However, for the game system to accurately know the position and posture of these props in real space, precise spatial calibration is essential. The prop spatial calibration scheme aims to establish an accurate coordinate correspondence between game props in the actual physical space and the virtual game space, enabling the game to respond in real-time and accurately when the player manipulates the props, thus enhancing the game's realism and enjoyment.

[0003] Common motion-sensing interaction devices on the market, such as the "Microsoft Kinect sensor and its effect," mainly combine RGB and depth cameras to capture the 3D skeleton of the human body to achieve virtual-real interaction. However, the lack of physical props inevitably increases the cost of game tutorials. Furthermore, due to hardware limitations, it has certain requirements for the usage environment; in complex backgrounds or when multiple people use it simultaneously, inaccurate recognition or interference may occur. The Nintendo Switch Joy-Con controllers, traditional game controllers, incorporate gyroscopes and accelerometer sensors to sense changes in the controller's posture, but cannot determine its precise position in space.

[0004] For example, the project "Keeping up with video game technology: Objective analysis of Xbox Kinect™ and PlayStation 3 Move™ for use in burn rehabilitation" utilizes a stereo system with Sony Eye to identify the position of props in the camera coordinate system. The PS Move spatial calibration method involves using a calibration mat called a "Calibration Mat." This mat is placed within the field of view of all PS Move tracking cameras, and then the PS Move controller is placed at each numbered position on the mat. The calibration tool records the pixel position of the PSMove bulb at each numbered position. Since the dimensions of the calibration mat are known (8.5 inches x 11 inches), we can define the center of the mat as the origin of the PS Move tracking space. By measuring the pixel positions, predefined X, Y, Z positions, and the focal length of each tracking camera, the position and orientation of the tracking camera relative to the tracking mat can be calculated. The overall calibration process is complex and time-consuming. "HTC Vive: Analysis and accuracy improvement." The Vive Tracker was developed that can be fixed to ordinary props. Combined with HTC Lighthouse, it can accurately locate the position and posture of props in physical space, allowing users to move freely in the virtual world. However, its relatively high price has limited its popularity.

[0005] To address the aforementioned technical problems, this invention proposes an efficient motion-sensing interactive prop and its physical space calibration method. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing an efficient motion-sensing interactive prop and its physical space calibration method.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for physical spatial calibration of an efficient motion-sensing interactive prop includes:

[0009] S1. Obtain the camera's intrinsic parameters, the extrinsic parameters between cameras, and the extrinsic parameters between the camera and the monitor;

[0010] S2. Collect data on the movement of the interactive prop; the data on the movement includes images of the light-emitting points in the interactive prop captured by the camera at different times and the corresponding first timestamp, the attitude data output by the cascaded inertial sensor (IMU) in the interactive prop and the corresponding second timestamp;

[0011] S3. Calculate the three-dimensional coordinate position of the light-emitting point using the camera intrinsic parameters and the extrinsic parameters between the camera and the display, and transform the three-dimensional coordinate position of the light-emitting point to a spatial position with the center of the display as the center of the world coordinate system, thus obtaining the spatial position of the light-emitting point in the display's world coordinate system.

[0012] S4. Interpolate the attitude data of the cascaded inertial sensor IMU to the three-dimensional coordinate data of the light source according to the second timestamp to obtain the attitude data of the cascaded inertial sensor IMU after interpolation;

[0013] S5. Based on the spatial position of the light-emitting point in the world coordinate system of the display and the attitude data after interpolation by the cascaded inertial sensor IMU, construct a set of nonlinear equations and solve the set of nonlinear equations to obtain the initial position, initial attitude of the interactive prop and the relative coordinates of the light-emitting point in space.

[0014] Furthermore, step S3 specifically includes:

[0015] S31. Using the camera's intrinsic parameters, back-project the pixel coordinates of the luminous points in the interactive props on the camera image plane to the ray direction vector in the camera coordinate system;

[0016] S32. Transform the ray direction vectors of different cameras to a unified coordinate system using the extrinsic parameters between cameras;

[0017] S33. Calculate the three-dimensional coordinates of the luminous point, and transform the three-dimensional coordinates to the spatial position in the world coordinate system of the display through the extrinsic parameters between the camera and the display.

[0018] Furthermore, the ray direction vector projected onto the camera coordinate system in step S31 is expressed as:

[0019] ;

[0020] Where, d i K represents the ray direction vector of the i-th camera; i Denotes the intrinsic parameter matrix of the i-th camera; m i This represents the normalized coordinates of the corresponding normalized pixel on the image plane of the i-th camera.

[0021] Furthermore, in step S32, the ray direction vectors of different cameras are transformed to a unified coordinate system using extrinsic parameters between the cameras, as shown below:

[0022] ;

[0023] in, Let represent the coordinate system of the ray direction vector of the i-th camera under the 1-th camera.

[0024] Furthermore, step S33 specifically includes:

[0025] S331. For each camera, establish a system of equations based on the geometric relationship between the ray direction vector and the emitting point;

[0026] S332. Solve the system of equations to obtain the three-dimensional coordinates of the light-emitting point. Transform the three-dimensional coordinates to the display world coordinate system using the extrinsic parameters between the camera and the display to obtain the spatial position of the light-emitting point in the display world coordinate system.

[0027] Furthermore, in step S4, the interpolation of the attitude data of the cascaded inertial sensor (IMU) to the three-dimensional coordinate data of the light-emitting point according to the second timestamp is performed using spherical linear interpolation.

[0028] Furthermore, step S5 specifically includes:

[0029] S51. Construct an error function based on the spatial position of the light-emitting point in the world coordinate system of the display and the attitude data after interpolation by the cascaded inertial sensor IMU;

[0030] S52. Taking the partial derivative of the error function yields a system of nonlinear equations;

[0031] S53. Use an optimization algorithm to iteratively solve the nonlinear equations to obtain the initial position, initial posture, and relative coordinates of the luminous point in space of the interactive prop.

[0032] Furthermore, the error function E in step S51 is expressed as:

[0033] ;

[0034] in,( x 0, y 0, z 0) indicates the initial position of the interactive item in the world coordinate system; R 0 indicates the initial posture of the interactive prop; (x local ,y local ,z local () represents the relative coordinates of the emitting point in space; x k , y k , z k ) represents the coordinates of the emitting point in world coordinates; This represents the rotation matrix from the room coordinate system to the geomagnetic coordinate system; This indicates that the cascaded inertial sensor (IMU) is in the first... k The rotation matrix under each timestamp is transformed to the room coordinate system; i represents the camera number; n represents the number of cameras.

[0035] Correspondingly, a highly efficient motion-sensing interactive prop is also provided, including an infrared LED light and a cascaded inertial sensor, with the infrared LED light and the cascaded inertial sensor rigidly connected.

[0036] Furthermore, it also includes several cameras and displays, the processing devices of which are connected to cascaded inertial sensors and cameras; the processing devices of the displays are used to execute the physical space calibration method for the efficient somatosensory interactive prop.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] 1. By combining the visual positioning of infrared LED lights with the attitude information from the IMU module, precise positioning and attitude estimation of interactive props in physical space can be achieved. This multi-sensor fusion method can effectively improve positioning accuracy and reduce errors. By simultaneously observing the light source with multiple cameras and calculating and fusing the observation data from different cameras, the positioning accuracy and robustness are further improved, enabling better handling of complex environments and multi-target situations.

[0039] 2. This solution can respond to player actions in real time and with precision, allowing the game to provide timely feedback and responses when players use interactive items, thus enhancing the game's interactivity and realism. Through precise spatial calibration, the player's actions and the movement of items in the virtual world can be highly consistent with the actions and movements of items in the real world, thereby enhancing the player's immersion and allowing them to more naturally integrate into the game environment.

[0040] 3. Through optimized algorithm and process design, the calibration process can be simplified, calibration time and complexity can be reduced, and calibration costs and difficulties can be lowered.

[0041] 4. This solution maintains high positioning accuracy and stability even in complex background environments, such as those with changing lighting or occlusion, demonstrating strong robustness. It can simultaneously handle the spatial calibration of multiple interactive props, making it suitable for motion-sensing game scenarios with multiple participants, thus improving the system's applicability and flexibility. Attached Figure Description

[0042] Figure 1 This is a structural diagram of an efficient motion-sensing interactive prop provided in the embodiment;

[0043] Figure 2 This is a flowchart of a method for physical space calibration of an efficient motion-sensing interactive prop provided in the embodiment;

[0044] Figure 3 This is a schematic diagram of timestamp interpolation provided in the embodiment. Detailed Implementation

[0045] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0046] The purpose of this invention is to address the shortcomings of existing technologies by providing an efficient motion-sensing interactive prop and its physical space calibration method.

[0047] Example 1

[0048] This embodiment provides an efficient motion-sensing interaction tool, such as... Figure 1 As shown, it includes an LED light 100, a cascaded inertial sensor 200, several cameras 300, and a display 400; the cameras 300 are infrared cameras; the cascaded inertial sensor 200 will be referred to as the IMU module.

[0049] Infrared LED 100 is rigidly connected to cascaded inertial sensor 200. These two can be approximated as a rigid body with a point feature. Camera 300 can be wired to the display processing device via Ethernet or USB to ensure image transmission speed; IMU data from cascaded inertial sensor 200 can be connected to the display processing device via Wi-Fi or Bluetooth to ensure stable data acquisition and transmission for subsequent multi-sensor data fusion.

[0050] In this embodiment, the infrared LED light's position in space can be acquired by several externally mounted cameras 300. A cascaded inertial sensor can provide the interactive prop with its pose in a geomagnetic coordinate system. For example... Figure 1 The diagram shows the spatial coordinate calibration of interactive props. This is achieved by calibrating the intrinsic parameters of known cameras and the extrinsic parameters between cameras. External parameters between camera and monitor The luminous point is combined with the camera's visual positioning. The attitude information of the cascaded inertial sensor 200 is used to solve the position and attitude of the prop in the display coordinate system. .

[0051] This embodiment combines the visual positioning of infrared LEDs with the attitude information of the IMU module to achieve accurate positioning and attitude estimation of interactive props in physical space.

[0052] Example 2

[0053] This implementation provides a method for efficient physical space calibration of motion-sensing interactive props, such as... Figure 2 As shown, this calibration method is based on a highly efficient motion-sensing interactive prop from Embodiment 1.

[0054] Calibration methods include:

[0055] S1. Obtain the camera's intrinsic parameters, the extrinsic parameters between cameras, and the extrinsic parameters between the camera and the monitor;

[0056] S2. Collect data on the movement of the interactive prop; the data on the movement includes images of the light-emitting points in the interactive prop captured by the camera at different times and the corresponding first timestamp, the attitude data output by the cascaded inertial sensor (IMU) in the interactive prop and the corresponding second timestamp;

[0057] S3. Calculate the three-dimensional coordinate position of the light-emitting point using the camera intrinsic parameters and the extrinsic parameters between the camera and the display, and transform the three-dimensional coordinate position of the light-emitting point to a spatial position with the center of the display as the center of the world coordinate system, thus obtaining the spatial position of the light-emitting point in the display's world coordinate system.

[0058] S4. Interpolate the attitude data of the cascaded inertial sensor IMU to the three-dimensional coordinate data of the light source according to the second timestamp to obtain the attitude data of the cascaded inertial sensor IMU after interpolation;

[0059] S5. Based on the spatial position of the light-emitting point in the world coordinate system of the display and the attitude data after interpolation by the cascaded inertial sensor IMU, construct a set of nonlinear equations and solve the set of nonlinear equations to obtain the initial position, initial attitude of the interactive prop and the relative coordinates of the light-emitting point in space.

[0060] In step S1, the intrinsic parameters of the camera, the extrinsic parameters between cameras, and the extrinsic parameters between the camera and the display are obtained.

[0061] The calibration of camera intrinsic parameters and extrinsic parameters between cameras can be achieved using a calibration plate or calibration rod, thereby obtaining the camera's intrinsic parameters and the extrinsic parameters between cameras; the extrinsic parameters between cameras only need to ensure that multiple cameras share a common field of view. However, there may be no shared field of view between the camera and the monitor, so calibration can be achieved using a front-coated reflector, thereby obtaining the extrinsic parameters between the camera and the monitor.

[0062] The front coated mirror can be found in the paper "A New Mirror-based Extrinsic Camera Calibration Using an Orthogonality Constraint", which will not be elaborated on in this embodiment.

[0063] In step S2, data on the movement of the interactive prop is collected; the data on the movement includes images of the light-emitting points in the interactive prop captured by the camera at different times and the corresponding first timestamp, attitude data output by the cascaded inertial sensor (IMU) in the interactive prop and the corresponding second timestamp.

[0064] In this embodiment, it is first necessary to ensure that the IMU module and the camera are working properly, and to collect data on the movement of the interactive prop (point feature rigid body). This data includes images of the infrared LED light continuously captured by the camera at a frequency of 60Hz at different times and the first timestamp, and the attitude data output by the IMU at a frequency of 100Hz and the corresponding second timestamp are saved simultaneously.

[0065] It should be noted that the image captured by the camera and the first timestamp, the attitude data output by the IMU and the corresponding second timestamp can be based on the subject itself or existing implementations, which will not be described in detail in this embodiment.

[0066] In step S3, the three-dimensional coordinate position of the light-emitting point is calculated using the camera intrinsic parameters and the extrinsic parameters between the camera and the display. The three-dimensional coordinate position of the light-emitting point is then transformed into a spatial position with the center of the display as the center of the world coordinate system, thus obtaining the spatial position of the light-emitting point in the display's world coordinate system.

[0067] By combining known camera intrinsic data and camera and display extrinsic parameters with image algorithms, the three-dimensional coordinate position of the luminous point is calculated and transformed into a spatial position with the center of the display screen as the center of the world coordinate system. Figure 1 The example shown has three cameras, but it is not limited to this.

[0068] The image algorithm can use the Hough transform detection circle algorithm in the open-source computer vision and machine learning software library OpenCV, which will not be elaborated on in this embodiment.

[0069] This embodiment assumes that there are n (n≥2) cameras observing the same infrared LED light emitting a single light source P in the space, which is also a three-dimensional point. Therefore, step S3 specifically involves:

[0070] S31. Using the camera's intrinsic parameters, back-project the pixel coordinates of the luminous points in the interactive props on the camera image plane to the ray direction vector in the camera coordinate system;

[0071] Camera intrinsic parameters include the camera's focal length, principal point coordinates, and distortion coefficients. These parameters can be calibrated using tools such as calibration boards or calibration rods. For the i-th camera (i∈1, 2, ...n), the intrinsic parameter matrix is ​​denoted as K. i It can project points in the camera coordinate system onto the corresponding image plane.

[0072] For each camera i, first use the intrinsic parameter matrix K i The ray direction vector after back-projecting the pixel coordinates onto the camera coordinate system is represented as:

[0073] ;

[0074] Where, d i K represents the ray direction vector of the i-th camera; i Denotes the intrinsic parameter matrix of the i-th camera; m i This represents the normalized coordinates of the corresponding normalized pixel on the image plane of the i-th camera.

[0075] The ray direction vector after reflection from each of the n cameras is calculated using the above formula.

[0076] S32. Transform the ray direction vectors of different cameras to a unified coordinate system using the extrinsic parameters between cameras;

[0077] The extrinsic parameter relationship between cameras is given by R, where the rotation matrix from camera i to camera j is denoted as R. ji Let t be the translation vector from camera i to camera j. ji , is used to transform the light source between different camera coordinate systems; j∈1, 2, ...n, where at most one camera is repeated. For example, taking the first camera as the reference, the rotation from 1,2...n to the first camera will only result in the case of equal values ​​from the first camera to the first camera. In other cases, i and j are different.

[0078] Let the coordinates of the luminous point P be X = (x, y, z). T The normalized coordinates of the corresponding normalized pixel on the image plane of the i-th camera are m. i =(u i ,v i ,1) T Where (x, y, z) represents the world coordinates of the luminous point P; (u i ,v i ,1) represents the coordinates of the normalized pixel point corresponding to the image plane of the i-th camera.

[0079] This embodiment utilizes the extrinsic parameters between cameras to transform all ray directions into a unified coordinate system. For example, the coordinate system of the ray direction vector of the first camera is obtained through step S31, and assuming that the coordinate system of the first camera is the basic coordinate system, then the ray direction vector of the i-th (i∈1,2,3,...n) camera is represented in the coordinate system of the first camera as follows:

[0080] ;

[0081] in, Let R represent the coordinate system of the ray direction vector of the i-th (i∈1,2,3,...n) camera under the first camera, and the rotation matrix from the first camera to the first camera is the identity matrix; 1i d represents the rotation matrix from the i-th camera to the 1-th camera; i Let represent the ray direction vector of the i-th (i∈1,2,3,...n) camera.

[0082] S33. Calculate the three-dimensional coordinates of the luminous point, and transform the three-dimensional coordinates to the spatial position in the world coordinate system of the display through the extrinsic parameters between the camera and the display.

[0083] S331. For each camera, establish a set of equations based on the ray direction vector calculated in step S31, the unified coordinate system transformed in step S32, and the geometric relationship of the light-emitting points; where the geometric relationship of the light-emitting points is that the imaging points in all cameras are the same three-dimensional light-emitting point P, then the line connecting the center of all cameras to the light-emitting point in the image plane of that camera must intersect at the same point and the light-emitting point P in three-dimensional space.

[0084] In the coordinate system of the first camera, assuming the coordinates of the emitting point (i.e., the 3D point) P are X1, based on the geometric relationship between all rays and the emitting point P, the following equations can be established for each camera i (i∈1, 2, ...n), and the system of equations is expressed as:

[0085] ;

[0086] Based on the geometric property that the cross product of two collinear vectors is 0, we obtain:

[0087] ;

[0088] Where X1 represents the coordinates of the light source P in the first camera; d represents the vector in the coordinate system of the ray direction vector of the i-th (i∈1,2,3,...n) camera under the first camera; i Let t represent the ray direction vector of the i-th camera (i∈1,2,3,...n); 1i μ represents the translation vector from the i-th camera (i∈2,3,...n) to the 1st camera; i Vector parameters representing the ray vector.

[0089] Rearranging according to the x, y, and z components of X1, it can be represented as follows:

[0090] ;

[0091] in, They represent Similarly, the components along the x, y, and z axes... , , Also t 1i Components on three axes.

[0092] Extract x, y, z from the above formula and rewrite it as follows: The linear equation, where It is a 3x3 matrix, represented as:

[0093] ;

[0094] A 3x1 matrix is ​​represented as:

[0095] ;

[0096] Given a total of n cameras, the system of linear equations for all parameter combinations is as follows:

[0097] ;

[0098] Right now:

[0099] A X1=b;

[0100] Where A represents A 1, A 2, ... A n The matrix formed by combining elements; b represents b 1, b 2, ... b n The matrix formed.

[0101] S332. Solve the system of equations to obtain the three-dimensional coordinates of the light-emitting point. Transform the three-dimensional coordinates to the display world coordinate system using the extrinsic parameters between the camera and the display to obtain the spatial position of the light-emitting point in the display world coordinate system.

[0102] Using the above system of equations, the coordinates X1 of the luminous point P are solved using methods such as SVD decomposition (but not limited to these). Here, a three-dimensional point has one and only one unique spatial coordinate; the subscript 1 is used because the first camera is taken as the reference coordinate. Then, the extrinsic parameter T between the camera and the display is used... c1Toscr Transform the coordinates to the monitor coordinate system to obtain the physical space point X of the light source point P in the monitor coordinate system. scr .

[0103] X scr = T c1Toscr * X1

[0104] This embodiment ultimately obtains the three-dimensional coordinate position of the infrared LED in the display coordinate system. This positional information will be used for subsequent multi-sensor data fusion and spatial calibration to achieve precise positioning and attitude calculation of the interactive prop.

[0105] In step S4, the attitude data of the cascaded inertial sensor IMU is interpolated to the three-dimensional coordinate data of the light source according to the second timestamp to obtain the attitude data of the cascaded inertial sensor IMU after interpolation.

[0106] The IMU outputs attitude data, including acceleration and angular velocity, at a frequency of 100Hz and records a second timestamp for each data point; the camera captures the position of the infrared LED at a frequency of 60Hz and records a first timestamp for each position; this embodiment needs to ensure that the timestamps of the IMU data and the LED coordinate data correspond to each other for subsequent interpolation processing, specifically:

[0107] The high-frequency IMU attitude data is interpolated onto the luminous point coordinate data according to the second timestamp, ensuring that corresponding IMU attitude data exists at each timestamp of the luminous point. This provides subsequent steps with attitude information of the luminous point at different timestamps, enabling more accurate spatial calibration. Figure 3 This diagram illustrates timestamp interpolation. The horizontal axis represents the time axis, the upper vertical line represents the timestamps of the IMU on the continuous time axis, and the lower vertical line represents the sampling timestamps of the IMG on the continuous time axis. The IMU data needs to be interpolated to the corresponding IMG timestamps. In this embodiment, the IMU attitude data is aligned with the luminous point position data, providing the necessary timestamp correspondence for subsequent solving of the nonlinear equations. Because the IMU output data is attitude data, while linear interpolation is simple and easy to implement, it is not suitable for scenarios with uneven rotation speeds. Therefore, this embodiment uses Spherical Linear Interpolation (SLERP). SLERP is a method of interpolation in quaternion space that maintains the smoothness and consistency of rotation. It is implemented by performing linear interpolation on a quaternion sphere.

[0108] For two IMU attitude data quaternions, assuming the two quaternions are q1 and q2, and the angle between the two quaternions is θ, it can be obtained by the dot product of the quaternions:

[0109] cosθ=q1*q2

[0110] Based on the spherical linear interpolation SLERP formula, the interpolation quaternion formula q(k) at time k is calculated as follows:

[0111] ;

[0112] Assuming the image timestamp is k, find the nearest previous and next IMU data points, each with timestamp k. q-1 and k q Then td = (k) q - k ) / (k q - k q-1 ) .

[0113] Where td represents the interpolation ratio, and represents the interpolation position from q1 to q2, which is the difference between the image IMG timestamp k and the nearest next IMU timestamp divided by the period of the IMU data.

[0114] For each luminous point coordinate timestamp, SLERP interpolation is used to calculate the corresponding IMU attitude data; and the interpolated IMU attitude data is aligned with the luminous point coordinate data to ensure that each luminous point coordinate has a precise attitude data corresponding to it.

[0115] This embodiment uses SLERP interpolation to obtain IMU attitude data at the timestamp of each emitting point coordinate. This data will be used for subsequent spatial calibration and attitude calculation to achieve accurate positioning and attitude estimation of the interactive prop; moreover, this embodiment can effectively combine high-frequency IMU attitude data with emitting point coordinate data to provide accurate attitude information for subsequent spatial calibration, thereby improving the positioning accuracy and attitude estimation accuracy of the interactive prop in physical space.

[0116] In step S5, a set of nonlinear equations is constructed based on the spatial position of the light-emitting point in the world coordinate system of the display and the attitude data after interpolation by the cascaded inertial sensor IMU. The nonlinear equations are then solved to obtain the initial position, initial attitude of the interactive prop, and the relative coordinates of the light-emitting point in space.

[0117] S51. Construct an error function based on the spatial position of the light-emitting point in the world coordinate system of the display and the attitude data after interpolation by the cascaded inertial sensor IMU;

[0118] Light emission point location data: Obtained from step S3 at offline time points k1, k2, ..., k n The three-dimensional coordinate position (x, y) of the luminous point in the display coordinate system obtained by constant observation k ,y k ,z k ), that is, X scr .

[0119] IMU attitude data: IMU attitude data obtained from interpolation in step S4, representing the attitude information of the interactive prop at different timestamps, usually expressed as a rotation matrix R. i It can be represented in quaternion form, and the two can be converted to each other.

[0120] Assume that at the initial moment, the position of the luminous point feature rigid body k1 in the world coordinate system is (x0, y0, z0) (initial translation vector), and the initial attitude is represented by the rotation matrix R0. Because the luminous point and the IMU module are rigidly connected, the coordinates of the luminous point in the rigid body's own coordinate system are assumed to be fixed (x0, y0, z0). local ,y local ,z local Simultaneously, define the rotation matrix from the room coordinate system to the geomagnetic coordinate system. .

[0121] For each observation time td k (k∈1,2,3,...m), the coordinates of the luminous point captured by the camera in the room coordinate system are (x k ,y k ,z k The IMU provides rotation information of the rigid body relative to the geomagnetic coordinate system (represented by a rotation matrix). First, the rotation information of the IMU is transformed into the room coordinate system. This is done through rotation matrix multiplication:

[0122]

[0123] Then, based on the coordinate transformation relationship, the coordinates of the luminous point in the room coordinate system can be expressed as:

[0124]

[0125] Expanding, we get:

[0126]

[0127] Based on the extended coordinate transformation relationship and multiple observation data, an error function E is constructed to measure the difference between the observed luminous point position and the theoretical position, expressed as:

[0128]

[0129] in,( x 0, y 0, z 0) indicates the initial position of the interactive item in the world coordinate system; R 0 indicates the initial posture of the interactive prop; (x local ,y local ,z local () represents the relative coordinates of the emitting point in space; x k , y k , z k) represents the coordinates of the emitting point in world coordinates; This represents the rotation matrix from the room coordinate system to the geomagnetic coordinate system; This indicates that the cascaded inertial sensor (IMU) is in the first... k The rotation matrix under each timestamp is transformed to the room coordinate system; i represents the camera number; n represents the number of cameras; 11 represents the first row and first column of the matrix, 12 represents the first row and second column of the matrix, 13 represents the first row and third column of the matrix, 21 represents the second row and first column of the matrix, 22 represents the second row and second column of the matrix, 23 represents the second row and third column of the matrix, 31 represents the third row and first column of the matrix, 32 represents the third row and second column of the matrix, and 33 represents the third row and third column of the matrix.

[0130] S52. Taking the partial derivative of the error function yields a system of nonlinear equations;

[0131] For the error function with respect to x 0, y 0, z 0, x local ,y local ,z local , Taking the partial derivatives and finding them to be zero yields a system of nonlinear equations. These equations can be computed using efficient nonlinear optimization libraries such as Ceres Solver.

[0132] S53. Use an optimization algorithm to iteratively solve the nonlinear equations to obtain the initial position, initial posture, and relative coordinates of the luminous point in space of the interactive prop.

[0133] An optimization algorithm is used to iteratively solve the problem. Multiple iterations minimize the error function, thus obtaining the solution. and IMU output And the position of the luminous point calculated by the visual algorithm (x k ,y k ,z k At any given moment, the position and orientation of the interactive prop in space can be obtained, enabling precise calibration of the interactive prop in physical space. This means that the initial position, initial orientation, and relative coordinates of the luminous point in the rigid body coordinate system of the interactive prop can be obtained.

[0134] The optimization algorithm can be implemented using the method described in "The Levenberg-Marquardt algorithm," specifically as follows:

[0135] Initial estimate: Provide a reasonable initial estimate, such as assuming the initial position and attitude are zero or estimating based on prior knowledge.

[0136] Iterative process: In each iteration, the value and gradient of the error function are calculated based on the current estimate, and the estimate is updated until the error function converges to a smaller value.

[0137] Output: The final result is the initial position of the interactive prop. x 0, y 0, z 0), initial posture R0, and relative coordinates (x, y) of the luminous point in the rigid body coordinate system. local ,y local ,z local ).

[0138] This embodiment achieves precise positioning and attitude estimation of interactive props in physical space by solving a system of nonlinear equations, thereby increasing the interactivity and playability of motion-sensing games.

[0139] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0140] 1. By combining the visual positioning of infrared LED lights with the attitude information from the IMU module, precise positioning and attitude estimation of interactive props in physical space can be achieved. This multi-sensor fusion method can effectively improve positioning accuracy and reduce errors. By simultaneously observing the light source with multiple cameras and calculating and fusing the observation data from different cameras, the positioning accuracy and robustness are further improved, enabling better handling of complex environments and multi-target situations.

[0141] 2. This solution can respond to player actions in real time and with precision, allowing the game to provide timely feedback and responses when players use interactive items, thus enhancing the game's interactivity and realism. Through precise spatial calibration, the player's actions and the movement of items in the virtual world can be highly consistent with the actions and movements of items in the real world, thereby enhancing the player's immersion and allowing them to more naturally integrate into the game environment.

[0142] 3. Through optimized algorithm and process design, the calibration process can be simplified, calibration time and complexity can be reduced, and calibration costs and difficulties can be lowered.

[0143] 4. This solution maintains high positioning accuracy and stability even in complex background environments, such as those with changing lighting or occlusion, demonstrating strong robustness. It can simultaneously handle the spatial calibration of multiple interactive props, making it suitable for motion-sensing game scenarios with multiple participants, thus improving the system's applicability and flexibility.

[0144] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for calibrating a physical space of a high-efficiency somatosensory interaction prop, characterized in that, The method comprises the following steps: S1. Obtain the internal parameters of the cameras, the external parameters between the cameras, and the external parameters between the cameras and the display; S2. Collect data in the motion of the interactive prop; the data in the motion comprises images of a light-emitting point in the interactive prop captured by the cameras at different time points and corresponding first time stamps, attitude data output by a cascaded inertial sensor IMU in the interactive prop and corresponding second time stamps; wherein the interactive prop comprises an infrared LED lamp and a cascaded inertial sensor, and the infrared LED lamp and the cascaded inertial sensor are rigidly connected; S3. Calculate the three-dimensional coordinate position of the light-emitting point by using the internal parameters of the cameras and the external parameters between the cameras and the display, and convert the three-dimensional coordinate position of the light-emitting point to a spatial position in a space with the center of the display as the center of the world coordinate system, to obtain the spatial position of the light-emitting point in the display world coordinate system; The step S3 is specifically as follows: S31. Project the pixel coordinates of the light-emitting point in the interactive prop on the camera image plane to a ray direction vector in the camera coordinate system by using the internal parameters of the cameras; S32. Convert the ray direction vectors of different cameras to a unified coordinate system by using the external parameters between the cameras; S33. Calculate the three-dimensional coordinates of the light-emitting point, and convert the three-dimensional coordinates to a spatial position in the display world coordinate system by using the external parameters between the cameras and the display; The step S33 is specifically as follows: S331. For each camera, establish an equation set according to the geometric relationship between the ray direction vector and the light-emitting point; S332. Solve the equation set to obtain the three-dimensional coordinates of the light-emitting point, and convert the three-dimensional coordinates to a spatial position in the display world coordinate system by using the external parameters between the cameras and the display, to obtain the spatial position of the light-emitting point in the display world coordinate system; S4. Interpolate the attitude data of the cascaded inertial sensor IMU to the three-dimensional coordinate data of the light-emitting point according to the second time stamps, to obtain the interpolated attitude data of the cascaded inertial sensor IMU; In the step S4, the attitude data of the cascaded inertial sensor IMU is interpolated to the three-dimensional coordinate data of the light-emitting point according to the second time stamps by using spherical linear interpolation; S5. Construct a nonlinear equation set according to the spatial position of the light-emitting point in the display world coordinate system and the interpolated attitude data of the cascaded inertial sensor IMU, and solve the nonlinear equation set to obtain the initial position, the initial attitude of the interactive prop and the relative coordinates of the light-emitting point in space; The step S5 is specifically as follows: S51. Construct an error function according to the spatial position of the light-emitting point in the display world coordinate system and the interpolated attitude data of the cascaded inertial sensor IMU; S52. Take the partial derivative of the error function to obtain a nonlinear equation set; S53. Use an optimization algorithm to iteratively solve the nonlinear equation set to obtain the initial position, the initial attitude of the interactive prop and the relative coordinates of the light-emitting point in space; The error function E of the step S51 is expressed as: E(x0,y0,z0,R0,x local ,y local ,z local , )= (( x k -( x 0+ · x local )) 2 +( y k -( y 0+ · y local )) 2 +( z k -( z 0+ · z local )) 2 ) wherein, x 0, y 0, z 0) represents the initial position of the interactive prop in the world coordinate system; R 0 represents the initial pose of the interactive prop;(x local ,y local ,z local ) represents the relative coordinates of the light emitting point in space; x k , y k , z k ) represents the coordinates of the light emitting point in the world coordinate; represents the rotation matrix from the room coordinate system to the geomagnetic coordinate system; represents the rotation matrix of the cascaded inertial sensor IMU at the i k th timestamp converted to the room coordinate system; i represents the number of the camera; n represents the number of the cameras.

2. The method of claim 1, wherein, The ray direction vector in the camera coordinate system in the step S31 is represented as: ; wherein d i represents the i-th camera ray direction vector; K i represents the i-th camera intrinsic matrix; m i represents the normalized coordinate of the corresponding normalized pixel point on the image plane of the i-th camera.

3. The method of claim 2, wherein, In the step S32, the ray direction vectors of different cameras are converted to a unified coordinate system by using the external parameters between the cameras, and are expressed as: ； wherein Table 1 Direction of the ray vector of the first camera in the coordinate system of the first camera.

4. A high-efficiency somatosensory interaction prop, characterized in that, The interactive prop comprises an infrared LED lamp and a cascaded inertial sensor, and the infrared LED lamp and the cascaded inertial sensor are rigidly connected; Also included are a plurality of cameras, a display, a processing device of the display, the processing device of the display being connected with the cascade inertial sensor, the camera; the processing device of the display is used for executing the physical space calibration method of the high-efficiency somatosensory interaction prop according to any one of claims 1-3.

Images