Efficient somatosensory interaction prop and physical space calibration method thereof

Abstract

The invention discloses an efficient somatosensory interaction prop and a physical space calibration method thereof, and the calibration method comprises the steps: S1, obtaining the internal parameters of cameras, the external parameters between the cameras, and the external parameters between the cameras and a display; s2, collecting data in interaction prop movement; s3, calculating the three-dimensional coordinate position of the light-emitting point through the internal reference of the camera and the external reference between the camera and the display, and converting the three-dimensional coordinate position of the light-emitting point to a spatial position taking the center of the display as the center of a world coordinate system to obtain the spatial position of the light-emitting point in the world coordinate system of the display; s4, interpolating the attitude data of the cascade inertial sensor IMU into the three-dimensional coordinate data of the luminous point according to a second timestamp to obtain the attitude data of the cascade inertial sensor IMU after interpolation; and S5, constructing a nonlinear equation set, and solving the nonlinear equation set to obtain the initial position and the initial attitude of the interactive prop and the relative coordinates of the luminous point in the space.

Description

Technical Field

[0001] The present invention relates to the field of virtual-reality interaction technology, and in particular to an efficient somatosensory interaction prop and a physical space calibration method thereof. Background Art

[0002] Somatosensory interactive games achieve interactive operations in the game by capturing the player's body movements or using the movement of special game props. However, in order for the game system to accurately know the position, posture and other information of these props in the real space, accurate spatial calibration must be performed. The prop space calibration solution aims to establish an accurate coordinate correspondence between the game props in the actual physical space and the virtual space of the game, so that when the player operates the props, the game can respond in real time and accurately, improving the realism and fun of the game.

[0003] Common somatosensory interactive props on the market, such as "Microsoft kinect sensor and its effect", mainly combine RGB cameras and depth cameras to capture the 3D skeleton of the human body to achieve virtual-real interaction. However, the lack of physical props will inevitably increase the cost of game teaching. At the same time, due to this hardware problem, there are certain requirements for the use environment. In complex background environments or when multiple people use it at the same time, inaccurate recognition or interference may occur. The Nintendo Switch Joy-Con, a traditional game controller, combines gyroscopes and accelerometer sensors inside, which can sense the changes in the controller's posture, but cannot determine its precise position in space.

[0004] Another example is "Keeping up with video game technology: Objective analysis ofXbox Kinect™ and PlayStation 3 Move™ for use in burn rehabilitation", which uses a binocular solution with Sony Eye to identify the position of props in the camera coordinate system. The spatial calibration method of PS Move involves the use of a calibration mat called a "Calibration Mat". This mat is placed in the field of view of all PS Move tracking cameras, and then the PS Move controller is placed at each numbered position on the mat. The calibration tool records the pixel position of the PS Move bulb at each numbered position. Since the size of the calibration mat is known (8.5 inches x 11 inches), we can define the center of the mat as the origin of the PS Move tracking space. By measuring the pixel position, the predefined X, Y, Z position, and the focal length of each tracking camera, the position and orientation of the tracking camera relative to the tracking mat can be calculated. The overall calibration process is complex and time-consuming. "HTC vive: Analysis and accuracy improvement." The Vive Tracker developed can be fixed on ordinary props. Combined with HTC lighthouse, it can accurately locate the position and posture of the props in the physical space, allowing users to move freely in the virtual world. However, the relatively high price limits its popularity.

[0005] In view of the above technical problems, the present invention proposes an efficient somatosensory interaction prop and a physical space calibration method thereof. Summary of the invention

[0006] The purpose of the present invention is to provide an efficient somatosensory interaction prop and a physical space calibration method thereof in view of the defects of the prior art.

[0007] In order to achieve the above objectives, the present invention adopts the following technical solutions: A physical space calibration method for an efficient somatosensory interactive prop, comprising: S1. Obtain the internal parameters of the camera, the external parameters between cameras, and the external parameters between the camera and the display; S2. Collecting data of the interactive props in motion; the data in motion includes the images of the light-emitting points in the interactive props continuously captured by the camera at different times and the corresponding first timestamps, the posture data output by the cascaded inertial sensor IMU in the interactive props and the corresponding second timestamps; S3. Calculate the three-dimensional coordinate position of the light-emitting point through the camera internal parameters and the external parameters between the camera and the display, and convert the three-dimensional coordinate position of the light-emitting point to a spatial position with the center of the display as the center of the world coordinate system, and obtain the spatial position of the light-emitting point in the world coordinate system of the display; S4. interpolating the posture data of the cascaded inertial sensor IMU to the three-dimensional coordinate data of the light-emitting point according to the second timestamp to obtain the interpolated posture data of the cascaded inertial sensor IMU; S5. According to the spatial position of the light-emitting point in the display world coordinate system and the interpolated posture data of the cascaded inertial sensor IMU, a nonlinear equation group is constructed and solved to obtain the initial position and initial posture of the interactive prop and the relative coordinates of the light-emitting point in space.

[0008] Furthermore, the step S3 is specifically as follows: S31. Back-project the pixel coordinates of the light-emitting point in the interactive prop on the camera image plane to the ray direction vector in the camera coordinate system through the camera's internal parameters; S32. Convert the ray direction vectors of different cameras to a unified coordinate system through the external parameters between the cameras; S33. Calculate the three-dimensional coordinates of the light-emitting point in the world coordinate system, and convert the three-dimensional coordinates to the spatial position in the world coordinate system of the display through the external reference between the camera and the display.

[0009] Furthermore, the ray direction vector back-projected to the camera coordinate system in step S31 is expressed as: ; Among them, d i Represents the ray direction vector of the i-th camera; K i represents the intrinsic parameter matrix of the i-th camera; m i Represents the normalized coordinates of the corresponding normalized pixel point on the image plane of the i-th camera.

[0010] Furthermore, in step S32, the ray direction vectors of different cameras are converted to a unified coordinate system through the external parameters between the cameras, which is expressed as: ; in, Represents the coordinate system of the ray direction vector of the i-th camera under the first camera.

[0011] Furthermore, the step S33 is specifically as follows: S331. For each camera, a set of equations is established according to the geometric relationship between the ray direction vector and the light emitting point; S332. Solve the equation group to obtain the three-dimensional coordinates of the light-emitting point in the world coordinate system. Through the external parameters between the camera and the display, the three-dimensional coordinates are converted to the display world coordinate system to obtain the spatial position of the light-emitting point in the display world coordinate system.

[0012] Furthermore, in step S4, interpolating the posture data of the cascaded inertial sensor IMU into the three-dimensional coordinate data of the light-emitting point according to the second timestamp is performed by using spherical linear interpolation.

[0013] Furthermore, the step S5 is specifically as follows: S51. Construct an error function according to the spatial position of the light-emitting point in the display world coordinate system and the interpolated attitude data of the cascaded inertial sensor IMU; S52. Calculate the partial derivative of the error function to obtain a nonlinear equation system; S53. Use the optimization algorithm to iteratively solve the nonlinear equation group to obtain the initial position, initial posture and relative coordinates of the light-emitting point of the interactive prop in space.

[0014] Furthermore, the error function E of step S51 is expressed as: ; in,( x 0, y 0, z 0) represents the initial position of the interactive prop in the world coordinate system; R 0 represents the initial posture of the interactive prop; (x local ,y local ,z local ) represents the relative coordinates of the light-emitting point in space; ( x k , y k , z k ) represents the coordinates of the light-emitting point in world coordinates; Represents the rotation matrix from the room coordinate system to the geomagnetic coordinate system; Indicates that the cascaded inertial sensor IMU is k The rotation matrix at a timestamp is converted to the room coordinate system; i represents the camera number; n represents the number of cameras.

[0015] Correspondingly, a highly efficient somatosensory interactive prop is also provided, including an infrared LED light and a cascade inertial sensor, and the infrared LED light and the cascade inertial sensor are rigidly connected.

[0016] Furthermore, it also includes several cameras and displays, and the processing devices of the displays are connected to the cascaded inertial sensors and cameras; the processing devices of the displays are used to execute the physical space calibration method of the efficient somatosensory interactive props.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. By combining the visual positioning of infrared LED lights and the attitude information of the IMU module, the interactive props can be accurately positioned and estimated in physical space. This multi-sensor fusion method can effectively improve the accuracy of positioning and reduce errors. By using multiple cameras to observe the light-emitting points at the same time, the accuracy and robustness of positioning are further improved by calculating and fusing the observation data of different cameras, which can better cope with complex environments and multi-target situations.

[0018] 2. This solution can respond to the player's operation in real time and accurately, so that when the player uses the interactive props, the game can give timely feedback and response, enhancing the interactivity and realism of the game. Through precise spatial calibration, the player's actions in the virtual world and the movement of props can be highly consistent with those in the real world, thereby enhancing the player's immersion and allowing the player to integrate into the game environment more naturally.

[0019] 3. Through optimized algorithms and process design, the calibration process can be simplified, the calibration time and complexity can be reduced, and the cost and difficulty of calibration can be reduced.

[0020] 4. This solution can maintain high positioning accuracy and stability in complex background environments, such as light changes, occlusions, etc., and has strong robustness. It can handle the spatial calibration of multiple interactive props at the same time, which is suitable for somatosensory game scenes with multiple people participating at the same time, and improves the applicability and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a structural diagram of an efficient somatosensory interaction prop provided by an embodiment; Figure 2 It is a flow chart of a physical space calibration method of an efficient somatosensory interaction prop provided by an embodiment; Figure 3 This is a schematic diagram of timestamp interpolation provided by an embodiment. DETAILED DESCRIPTION

[0022] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0023] The purpose of the present invention is to provide an efficient somatosensory interaction prop and a physical space calibration method thereof in view of the defects of the prior art.

[0024] Embodiment 1

[0025] This embodiment provides an efficient somatosensory interaction prop, such as Figure 1 As shown, it includes an LED lamp 100, a cascade inertial sensor 200, several cameras 300, and a display 400; the camera 300 is an infrared camera; the cascade inertial sensor 200 is hereinafter referred to as an IMU module.

[0026] The infrared LED lamp 100 is rigidly connected to the cascade inertial sensor 200. The two can be approximated as a rigid body with a point feature. The camera 300 can be wired to the display processing device through an Ethernet port or USB to ensure the image transmission rate; the IMU data in the cascade inertial sensor 200 can be connected to the display processing device through Wifi or Bluetooth to ensure the stability of data acquisition and transmission for subsequent multi-sensor data fusion.

[0027] In this embodiment, the infrared LED light can obtain its position in space through several cameras 300 built externally. The cascaded inertial sensor can provide the interactive prop with the position in the geomagnetic coordinate system. Figure 1 The figure shows the schematic diagram of the calibration of the interactive prop space coordinates. By calibrating the intrinsic parameters of the known camera and the extrinsic parameters between the cameras , external parameters between camera and display , light point combined with camera visual positioning , the attitude information of the cascade inertial sensor 200, and then solve the position and attitude of the prop in the display coordinate system .

[0028] This embodiment combines the visual positioning of the infrared LED light and the posture information of the IMU module to achieve accurate positioning and posture estimation of the interactive props in the physical space.

[0029] Embodiment 2

[0030] This implementation provides an efficient physical space calibration method for somatosensory interactive props, such as Figure 2As shown, the calibration method is implemented based on an efficient somatosensory interaction prop in the first embodiment.

[0031] Calibration methods include: S1. Obtain the internal parameters of the camera, the external parameters between cameras, and the external parameters between the camera and the display; S2. Collecting data of the interactive props in motion; the data in motion includes the images of the light-emitting points in the interactive props continuously captured by the camera at different times and the corresponding first timestamps, the posture data output by the cascaded inertial sensor IMU in the interactive props and the corresponding second timestamps; S3. Calculate the three-dimensional coordinate position of the light-emitting point through the camera internal parameters and the external parameters between the camera and the display, and convert the three-dimensional coordinate position of the light-emitting point to a spatial position with the center of the display as the center of the world coordinate system, and obtain the spatial position of the light-emitting point in the world coordinate system of the display; S4. interpolating the posture data of the cascaded inertial sensor IMU to the three-dimensional coordinate data of the light-emitting point according to the second timestamp to obtain the interpolated posture data of the cascaded inertial sensor IMU; S5. According to the spatial position of the light-emitting point in the display world coordinate system and the interpolated posture data of the cascaded inertial sensor IMU, a nonlinear equation group is constructed and solved to obtain the initial position and initial posture of the interactive prop and the relative coordinates of the light-emitting point in space.

[0032] In step S1, the intrinsic parameters of the camera, the extrinsic parameters between cameras, and the extrinsic parameters between the camera and the display are obtained.

[0033] The camera internal parameters and external parameters between cameras can be calibrated based on a calibration plate or a calibration rod to obtain the camera internal parameters and external parameters between cameras; the external parameters between cameras only need to ensure that multiple cameras have a common field of view. There may not be a common field of view between the camera and the display, so calibration can be achieved with the help of a front-coated reflector to obtain the external parameters between the camera and the display.

[0034] The front coated reflector can refer to the reference paper "A New Mirror-based Extrinsic Camera Calibration Using an Orthogonality Constraint", which will not be described in detail in this embodiment.

[0035] In step S2, data of the interactive prop in motion is collected; the data in motion includes images of the light-emitting points in the interactive props continuously captured by the camera at different times and the corresponding first timestamps, and posture data output by the cascaded inertial sensor IMU in the interactive props and the corresponding second timestamps.

[0036] In this embodiment, we must first ensure that the IMU module and the camera are working properly, and collect data on the interactive props (point feature rigid body) in motion. The data includes the images of the infrared LED light captured continuously at a frequency of 60Hz by the camera at different times and the first timestamp, and the posture data output by the IMU at a frequency of 100Hz and the corresponding second timestamp are synchronously saved.

[0037] It should be noted that the camera captures the image and the first timestamp, the posture data output by the IMU and the corresponding second timestamp can be based on the subject itself or an existing implementation, and will not be described in detail in this embodiment.

[0038] In step S3, the three-dimensional coordinate position of the light-emitting point is calculated through the camera intrinsic parameters and the external parameters between the camera and the display, and the three-dimensional coordinate position of the light-emitting point is converted to a spatial position with the center of the display as the center of the world coordinate system, so as to obtain the spatial position of the light-emitting point in the world coordinate system of the display.

[0039] The image algorithm combines the known camera intrinsic parameter data and the external parameters of the camera and display to calculate the three-dimensional coordinate position of the light-emitting point and convert it to a spatial position with the center of the display screen as the center of the world coordinate system. Figure 1 A case of three cameras is shown, but is not limited thereto.

[0040] The image algorithm may adopt the Hough transform circle detection algorithm in the open source computer vision and machine learning software library Opencv, which will not be described in detail in this embodiment.

[0041] This embodiment assumes that there are n (n≥2) cameras observing the same light point P emitted by an infrared LED lamp in the space, which is also a three-dimensional point. Therefore, step S3 is specifically as follows: S31. Back-project the pixel coordinates of the light-emitting point in the interactive prop on the camera image plane to the ray direction vector in the camera coordinate system through the camera's internal parameters; The camera internal parameters include the focal length, principal point coordinates and distortion coefficient of the camera. These parameters can be calibrated by tools such as calibration plates or calibration rods. For the i-th camera (i∈1, 2, ...n), the internal parameter matrix is ​​recorded as K i , which can project the points in the camera coordinate system onto the corresponding image plane.

[0042] For each camera i, first use the internal parameter matrix K i The pixel coordinates are back-projected to the ray direction vector in the camera coordinate system. Therefore, the ray direction vector after back-projection of the i-th camera is expressed as: ; Among them, d i Represents the ray direction vector of the i-th camera; Ki represents the intrinsic parameter matrix of the i-th camera; m i Represents the normalized coordinates of the corresponding normalized pixel point on the image plane of the i-th camera.

[0043] According to the above formula, the ray direction vector of each of the n cameras after backprojection is calculated.

[0044] S32. Convert the ray direction vectors of different cameras to a unified coordinate system through the external parameters between the cameras; The external parameter relationship between cameras, let the rotation matrix from camera i to camera j be recorded as R ji , the translation vector from camera i to camera j is denoted by t ji , used to convert the light-emitting point between different camera coordinate systems; j∈1, 2, ...n, where there is at most one repetition. For example, based on the first camera, the rotation from 1,2...n to the first camera will only be equal from the first camera to the first camera. In other cases, i and j are different.

[0045] Assume that the coordinates of the light emitting point P in the world coordinate system are X=(x,y,z) T , the normalized coordinates of the corresponding normalized pixel point on the image plane of the i-th camera are m i =(u i ,v i ,1) T . Where (x, y, z) represents the world coordinates of the light point P; (u i ,v i ,1) represents the coordinates of the corresponding normalized pixel point on the image plane of the i-th camera.

[0046] This embodiment uses the external parameters between cameras to convert all ray directions to a unified coordinate system. For example, the coordinate system of the ray direction vector of the first camera is obtained through step S31, and assuming that the coordinate system of the first camera is the basic coordinate system, the ray direction vector of the i-th (i∈1,2,3,...n) camera is expressed in the coordinate system of the first camera as: ; in, Represents the coordinate system of the ray direction vector of the i-th (i∈1,2,3,...n) camera under the first camera, and the rotation matrix from the first camera to the first camera is the unit matrix; R 1i Represents the rotation matrix from the i-th camera to the first camera; d i Represents the ray direction vector of the i-th (i∈1,2,3,...n) camera.

[0047] S33. Calculate the three-dimensional coordinates of the light-emitting point in the world coordinate system, and convert the three-dimensional coordinates to the spatial position in the world coordinate system of the display through the external reference between the camera and the display.

[0048] S331. For each camera, a set of equations is established based on the ray direction vector calculated in step S31, the unified coordinate system converted in step S32, and the geometric relationship of the light-emitting points; wherein the geometric relationship of the light-emitting points is that the imaging points in all cameras are the same three-dimensional light-emitting point P, and the lines connecting the centers of all cameras and the light-emitting points in the image plane of the camera must intersect with the same point and the light-emitting point P in the three-dimensional space.

[0049] In the coordinate system of the first camera, assuming that the coordinate of the light-emitting point (i.e., three-dimensional point) P is X1, according to the geometric relationship between all rays and the light-emitting point P, the following equation can be established for each camera i (i∈1, 2, ...n), and the equation system is expressed as: ; By geometric properties, the cross product of two vectors being collinear is 0, and we get: ; Among them, X1 represents the coordinates of the light-emitting point P in the first camera; The vector representing the ray direction vector of the i-th (i∈1,2,3,...n) camera in the coordinate system of the first camera; d i represents the ray direction vector of the i-th (i∈1,2,3,...n) camera; t 1i represents the translation vector from the i-th (i∈2,3,...n) camera to the first camera; μ i Vector parameter representing the ray vector.

[0050] Arrange according to the x, y, and z components of X1, expressed as: ; in, Respectively The components of the x, y, and z axes are similar. , , Also t 1i Components on three axes.

[0051] Extract x, y, z from the above formula and rewrite it as The linear equation of is a 3x3 matrix, represented as: ; is a 3x1 matrix, expressed as: ; There are n cameras in total, so the linear equations for all parameter combinations are: ; Right now: A X1=b; Among them, A represents A 1, A 2, ... A n The matrix formed by combining 1, b 2, ... b n The matrix composed of.

[0052] S332. Solve the equation group to obtain the three-dimensional coordinates of the light-emitting point in the world coordinate system. Through the external parameters between the camera and the display, the three-dimensional coordinates are converted to the display world coordinate system to obtain the spatial position of the light-emitting point in the display world coordinate system.

[0053] Through the above equations, the coordinates X1 of the light-emitting point P in the world coordinate system are solved using methods such as SVD decomposition (but not limited to this), where a three-dimensional point has only one unique spatial coordinate, and the subscript 1 is because the first camera is used as the reference coordinate. Then, the external parameter T between the camera and the display is used. c1Toscr , convert the coordinates to the display coordinate system, and obtain the physical space point X of the light emitting point P in the display coordinate system scr .

[0054] X scr = T c1Toscr * X1 This embodiment finally obtains the three-dimensional coordinate position of the infrared LED light in the display coordinate system. This position information will be used for subsequent multi-sensor data fusion and space calibration to achieve accurate positioning and posture calculation of the interactive props.

[0055] In step S4, the posture data of the cascaded inertial sensor IMU is interpolated to the three-dimensional coordinate data of the light-emitting point according to the second timestamp to obtain the interpolated posture data of the cascaded inertial sensor IMU.

[0056] The IMU outputs attitude data at a frequency of 100 Hz, including acceleration, angular velocity and other information, and records the second timestamp of each data point; the camera captures the position of the infrared LED light at a frequency of 60 Hz, and records the first timestamp of each position; this embodiment needs to ensure that the timestamps of the IMU data and the light-emitting point coordinate data can correspond to each other in order to perform subsequent interpolation processing, specifically: According to the second timestamp, the high-frequency IMU attitude data is interpolated to the light-emitting point coordinate data to ensure that the corresponding IMU attitude data exists at the timestamp corresponding to the light-emitting point, and the attitude information of the light-emitting point at different timestamps is provided for the subsequent steps to perform more accurate spatial calibration. Figure 3 It is a schematic diagram of timestamp interpolation. The middle horizontal axis is the time axis, the upper vertical line is the timestamp of the IMU on the continuous time axis, and the lower vertical line is the sampling timestamp of the IMG on the continuous time axis. The IMU data needs to be interpolated to the corresponding IMG timestamp. This embodiment aligns the IMU attitude data with the light point position data, providing the necessary timestamp correspondence for the subsequent solution of the nonlinear equations. Because the IMU output data is attitude data, if linear interpolation is used, although it is simple and easy to implement, it is not very suitable for scenarios with uneven rotation speeds; therefore, this embodiment uses spherical linear interpolation (SLERP). SLERP is a method for interpolation in quaternion space that can maintain the smoothness and consistency of rotation. It is achieved by linear interpolation on the quaternion sphere.

[0057] For two IMU attitude data quaternions, assuming that the two quaternions are q1 and q2, the angle between the two quaternions is θ, which can be obtained by the dot product of the quaternions: cosθ=q1*q2 According to the spherical linear interpolation SLERP formula, the interpolation quaternion formula q(k) at time k is calculated, which is expressed as: ; Assuming the image timestamp is time k, find the nearest previous and next IMU data, which are timestamp k respectively. q-1 and k q , then td = ( k q - k ) / (k q -k q-1 ) .

[0058] Among them, td represents the interpolation ratio, which represents the interpolation position from q1 to q2, that is, the difference between the image IMG timestamp k and the next nearest IMU timestamp divided by the period of the IMU data.

[0059] For the timestamp of each light-emitting point coordinate, the corresponding IMU attitude data is calculated using SLERP interpolation; and the interpolated IMU attitude data is aligned with the light-emitting point coordinate data to ensure that each light-emitting point coordinate has an accurate attitude data corresponding to it.

[0060] This embodiment obtains IMU attitude data at the timestamp of each light-emitting point coordinate through SLERP interpolation. These data will be used for subsequent spatial calibration and attitude calculation to achieve accurate positioning and attitude estimation of interactive props; and this embodiment can effectively combine high-frequency IMU attitude data with light-emitting point coordinate data to provide accurate attitude information for subsequent spatial calibration, thereby improving the positioning accuracy of interactive props in physical space and the accuracy of attitude estimation.

[0061] In step S5, a nonlinear equation group is constructed based on the spatial position of the light-emitting point in the display world coordinate system and the interpolated posture data of the cascaded inertial sensor IMU, and the nonlinear equation group is solved to obtain the initial position, initial posture and relative coordinates of the light-emitting point in space of the interactive prop.

[0062] S51. Construct an error function according to the spatial position of the light-emitting point in the display world coordinate system and the interpolated attitude data of the cascaded inertial sensor IMU; Light point position data: obtained from step S3 at offline time points k1, k2, ..., k n The three-dimensional coordinate position (x k ,y k ,z k ), that is, X scr .

[0063] IMU attitude data: The IMU attitude data interpolated from step S4 represents the attitude information of the interactive prop at different timestamps, usually in the form of a rotation matrix R i Or quaternion form, the two can be converted to each other.

[0064] Assume that at the initial moment of the light-emitting point feature rigid body k1, the position in the world coordinate system is (x0, y0, z0) (initial translation vector), and the initial posture is represented by the rotation matrix R0. Because the light-emitting point is rigidly connected to the IMU module, the coordinates of the light-emitting point in the rigid body's own coordinate system are assumed to be fixed (x local ,y local ,z local ). At the same time, set the rotation matrix from the room coordinate system to the geomagnetic coordinate system .

[0065] For each observation time td k (k∈1,2,3,...m), the camera captures the coordinates of the light point in the room coordinate system as (x k ,y k ,z k ), the IMU gives the rotation information of the rigid body relative to the geomagnetic coordinate system (expressed as the rotation matrix ). First, convert the IMU's rotation information to the room coordinate system, and then multiply the rotation matrix to get:

[0066] Then, according to the coordinate transformation relationship, the coordinates of the light point in the room coordinate system can be expressed as:

[0067] Expand to get:

[0068] Based on the above expanded coordinate transformation relationship and multiple observed data, an error function E is constructed to measure the difference between the observed luminous point position and the theoretical position, which is expressed as:

[0069] in,( x 0, y 0, z 0) represents the initial position of the interactive prop in the world coordinate system; R 0 represents the initial posture of the interactive prop; (x local ,y local ,z local ) represents the relative coordinates of the light-emitting point in space; ( x k , y k , z k ) represents the coordinates of the light-emitting point in world coordinates; Represents the rotation matrix from the room coordinate system to the geomagnetic coordinate system; Indicates that the cascaded inertial sensor IMU is k The rotation matrix at each timestamp is converted to the room coordinate system; i represents the camera number; n represents the number of cameras; 11 represents the first row and first column of the matrix, 12 represents the first row and second column of the matrix, 13 represents the first row and third column of the matrix, 21 represents the second row and first column of the matrix, 22 represents the second row and second column of the matrix, 23 represents the second row and third column of the matrix, 31 represents the third row and first column of the matrix, 32 represents the third row and second column of the matrix, and 33 represents the third row and third column of the matrix.

[0070] S52. Calculate the partial derivative of the error function to obtain a nonlinear equation system; The error function is respectively about x 0, y 0, z 0,x local ,y local ,z local , , find the partial derivative to be 0, and get a set of nonlinear equations. The obtained nonlinear equations can be calculated with the help of efficient nonlinear optimization libraries such as ceres solver.

[0071] S53. Use the optimization algorithm to iteratively solve the nonlinear equation group to obtain the initial position, initial posture and relative coordinates of the light-emitting point of the interactive prop in space.

[0072] Use the optimization algorithm to iterate and solve, and minimize the error function through multiple iterations to obtain And the IMU output And the position of the light point calculated by the visual algorithm (x k ,y k ,z k ), the position and posture of the interactive props in space can be obtained at any time, and the interactive props can be accurately calibrated in the physical space, that is, the initial position, initial posture and relative coordinates of the light-emitting point of the interactive props in the rigid body coordinate system can be obtained.

[0073] The optimization algorithm can be implemented by referring to "The levenberg-marquardt algorithm.", specifically: Initial estimate: Provide a reasonable initial estimate, such as assuming the initial position and attitude are zero or estimating based on prior knowledge.

[0074] Iterative process: In each iteration, the value and gradient of the error function are calculated based on the current estimate, and the estimate is updated until the error function converges to a smaller value.

[0075] Output result: The initial position of the interactive prop is finally obtained ( x 0, y 0, z 0), the initial posture R0, and the relative coordinates of the light-emitting point in the rigid body coordinate system (x local ,y local ,z local ).

[0076] This embodiment achieves accurate positioning and posture estimation of interactive props in physical space by solving a group of nonlinear equations, thereby increasing the interactivity and playability of the somatosensory game.

[0077] Compared with the prior art, the present invention has the following beneficial effects: 1. By combining the visual positioning of infrared LED lights and the attitude information of the IMU module, the interactive props can be accurately positioned and estimated in physical space. This multi-sensor fusion method can effectively improve the accuracy of positioning and reduce errors. By using multiple cameras to observe the light-emitting points at the same time, the accuracy and robustness of positioning are further improved by calculating and fusing the observation data of different cameras, which can better cope with complex environments and multi-target situations.

[0078] 2. This solution can respond to the player's operation in real time and accurately, so that when the player uses the interactive props, the game can give timely feedback and response, enhancing the interactivity and realism of the game. Through precise spatial calibration, the player's actions in the virtual world and the movement of props can be highly consistent with those in the real world, thereby enhancing the player's immersion and allowing the player to integrate into the game environment more naturally.

[0079] 3. Through optimized algorithms and process design, the calibration process can be simplified, the calibration time and complexity can be reduced, and the cost and difficulty of calibration can be reduced.

[0080] 4. This solution can maintain high positioning accuracy and stability in complex background environments, such as light changes, occlusions, etc., and has strong robustness. It can handle the spatial calibration of multiple interactive props at the same time, which is suitable for somatosensory game scenes with multiple people participating at the same time, and improves the applicability and flexibility of the system.

[0081] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A physical space calibration method for efficient somatosensory interactive props, characterized in that: include: S1. Obtain the internal parameters of the camera, the external parameters between cameras, and the external parameters between the camera and the display; S2. Collecting data of the interactive props in motion; the data in motion includes the images of the light-emitting points in the interactive props continuously captured by the camera at different times and the corresponding first timestamps, the posture data output by the cascaded inertial sensor IMU in the interactive props and the corresponding second timestamps; S3. Calculate the three-dimensional coordinate position of the light-emitting point through the camera internal parameters and the external parameters between the camera and the display, and convert the three-dimensional coordinate position of the light-emitting point to a spatial position with the center of the display as the center of the world coordinate system, and obtain the spatial position of the light-emitting point in the world coordinate system of the display; S4. interpolating the posture data of the cascaded inertial sensor IMU to the three-dimensional coordinate data of the light-emitting point according to the second timestamp to obtain the interpolated posture data of the cascaded inertial sensor IMU; S5. According to the spatial position of the light-emitting point in the display world coordinate system and the interpolated posture data of the cascaded inertial sensor IMU, a nonlinear equation group is constructed and solved to obtain the initial position and initial posture of the interactive prop and the relative coordinates of the light-emitting point in space.

2. According to claim 1, a physical space calibration method for an efficient somatosensory interactive prop is characterized in that: The step S3 is specifically as follows: S31. Back-project the pixel coordinates of the light-emitting point in the interactive prop on the camera image plane to the ray direction vector in the camera coordinate system through the camera's internal parameters; S32. Convert the ray direction vectors of different cameras to a unified coordinate system through the external parameters between the cameras; S33. Calculate the three-dimensional coordinates of the light-emitting point in the world coordinate system, and convert the three-dimensional coordinates to the spatial position in the world coordinate system of the display through the external reference between the camera and the display.

3. According to the method for calibrating the physical space of an efficient somatosensory interactive prop, the method is characterized in that: The ray direction vector back-projected to the camera coordinate system in step S31 is expressed as: ； Among them, d i Represents the ray direction vector of the i-th camera; K i represents the intrinsic parameter matrix of the i-th camera; m i Represents the normalized coordinates of the corresponding normalized pixel point on the image plane of the i-th camera.

4. According to claim 3, a physical space calibration method for efficient somatosensory interactive props is characterized in that: In step S32, the ray direction vectors of different cameras are converted to a unified coordinate system through the external parameters between the cameras, which is expressed as: ； in, Represents the coordinate system of the ray direction vector of the i-th camera under the first camera.

5. According to claim 4, a physical space calibration method for efficient somatosensory interactive props is characterized in that: The step S33 is specifically as follows: S331. For each camera, a set of equations is established according to the geometric relationship between the ray direction vector and the light emitting point; S332. Solve the equation group to obtain the three-dimensional coordinates of the light-emitting point in the world coordinate system. Through the external parameters between the camera and the display, the three-dimensional coordinates are converted to the display world coordinate system to obtain the spatial position of the light-emitting point in the display world coordinate system.

6. The physical space calibration method of an efficient somatosensory interactive prop according to claim 1, characterized in that: In the step S4, the posture data of the cascaded inertial sensor IMU is interpolated into the three-dimensional coordinate data of the light-emitting point according to the second timestamp by using spherical linear interpolation.

7. The method for calibrating a physical space of an efficient somatosensory interactive prop according to claim 1, characterized in that: The step S5 is specifically as follows: S51. Construct an error function according to the spatial position of the light-emitting point in the display world coordinate system and the interpolated attitude data of the cascaded inertial sensor IMU; S52. Calculate the partial derivative of the error function to obtain a nonlinear equation system; S53. Use the optimization algorithm to iteratively solve the nonlinear equation group to obtain the initial position, initial posture and relative coordinates of the light-emitting point of the interactive prop in space.

8. The method for calibrating the physical space of an efficient somatosensory interactive prop according to claim 7, characterized in that: The error function E of step S51 is expressed as: ； in,( x 0, y 0, z 0) represents the initial position of the interactive prop in the world coordinate system; R 0 represents the initial posture of the interactive prop; (x local ,y local ,z local ) represents the relative coordinates of the light-emitting point in space; ( x k , y k , z k ) represents the coordinates of the light-emitting point in world coordinates; Represents the rotation matrix from the room coordinate system to the geomagnetic coordinate system; Indicates that the cascaded inertial sensor IMU is k The rotation matrix at a timestamp is converted to the room coordinate system; i represents the camera number; n represents the number of cameras.

9. An efficient somatosensory interactive prop, characterized in that: It includes an infrared LED light and a cascade inertial sensor, and the infrared LED light and the cascade inertial sensor are rigidly connected.

10. The efficient somatosensory interactive prop according to claim 9, characterized in that: It also includes several cameras and displays, and the processing devices of the displays are connected to the cascaded inertial sensors and cameras; the processing devices of the displays are used to execute the physical space calibration method of an efficient somatosensory interactive prop as described in any one of claims 1-8.

Images