An apparatus based on virtual-real interaction and its pose calculation method

Abstract

The present invention discloses a virtual-real interaction-based device and its pose calculation method. A pose calculation method for a virtual-real interaction device involves: S1. Obtaining the coordinates of the projection emitted by a laser in a preset virtual plane coordinate system to obtain a first coordinate; S2. Obtaining the spot image of the laser emitted to the projection plane collected by a camera, and processing the obtained spot image to obtain the image coordinates of the spot; S3. Obtaining the internal parameters and external parameters of the camera, and converting the image coordinates of the spot into the coordinates in the projection plane coordinate system based on the internal parameters and external parameters of the camera to obtain a second coordinate; S4. Calculating the pose data of the virtual-real interaction device in the projection plane coordinate system according to the first coordinate and the second coordinate and based on the EPnP algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual - reality interaction, and particularly to a device based on virtual - reality interaction and a method for calculating its pose. Background Art

[0002] With the development of technologies such as virtual reality (VR) and augmented reality (AR), various forms of virtual - reality interaction games have become increasingly popular, and the demand for new interaction props is constantly growing. The interaction props in the prior art often lack accurate positioning and real - time interaction capabilities, which limits the improvement of the game experience.

[0003] In view of the above - mentioned technical problems, the present invention proposes a device based on virtual - reality interaction and a method for calculating its pose. Summary of the Invention

[0004] The purpose of the present invention is to provide a device based on virtual - reality interaction and a method for calculating its pose in view of the deficiencies of the prior art.

[0005] To achieve the above - mentioned purpose, the present invention adopts the following technical solutions:

[0006] A method for calculating the pose of a device based on virtual - reality interaction includes:

[0007] S1. Obtain the coordinates of the projection emitted by the laser in the preset virtual - plane coordinate system to obtain the first coordinates;

[0008] S2. Obtain the spot image of the laser emitted to the projection plane collected by the camera, and process the obtained spot image to obtain the image coordinates of the spot;

[0009] S3. Obtain the internal parameters and external parameters of the camera, and convert the image coordinates of the spot into the coordinates in the projection - plane coordinate system based on the internal parameters and external parameters of the camera to obtain the second coordinates;

[0010] S4. Calculate the pose data of the virtual - reality interaction device in the projection - plane coordinate system according to the first coordinates and the second coordinates and based on the EPnP algorithm.

[0011] Further, after the step S4, it further includes:

[0012] S5. Post - process the pose data and transmit the post - processed pose data to the host computer system.

[0013] Further, the step S3 is specifically:

[0014] S31. Calculate the direction vector of the spot in the camera coordinate system according to the image coordinates of the spot and the internal parameters of the camera;

[0015] S32. Obtain the extrinsic parameter matrix of the camera, and calculate the rotation component and translation component of the extrinsic parameter matrix;

[0016] S33. Based on the rotation component, convert the direction vector into a component in the projection plane coordinate system;

[0017] S34. According to the translation component and the converted component in step S33, calculate the coordinates of the light spot in the image coordinate system converted into the coordinates in the projection plane coordinate system to obtain the second coordinate.

[0018] Further, the direction vector of the light spot in the camera coordinate system in step S31 is expressed as:

[0019] ;

[0020] where, represents the direction vector of the light spot in the camera coordinate system; K represents the internal parameters of the camera; p i represents the image coordinates of the light spot; C represents the camera coordinate system.

[0021] Further, the translation component of the extrinsic parameter matrix in step S32 is expressed as:

[0022] ;

[0023] where, represents the translation component; G represents the projection plane coordinate system; t Cx represents the translation component of the x-axis; t Cy represents the translation component of the y-axis; t Cz represents the translation component of the z-axis; T represents matrix transpose.

[0024] Further, the component converted into the projection plane coordinate system in step S33 is expressed as:

[0025] ;

[0026] where, represents the component converted into the projection plane coordinate system; represents the rotation component of the extrinsic parameter matrix; d ix represents the component on the x-axis; d iy represents the component on the y-axis; d iz represents the component on the z-axis; T represents matrix transpose.

[0027] Further, the second coordinate in step S34 is expressed as:

[0028] ;

[0029] Among them, represents the second coordinate, that is, the image coordinates of the light spot are converted into the coordinates in the projection plane coordinate system.

[0030] Correspondingly, a virtual-real interaction device is also provided, including a virtual-real interaction body, a calculation unit, and a camera. The virtual-real interaction body includes a housing and an inertial measurement unit and several lasers arranged in the housing. The several lasers are all connected to the inertial measurement unit, and the calculation unit is connected to the inertial measurement unit and the camera. The calculation unit is used to execute the pose calculation method of a virtual-real interaction device.

[0031] Furthermore, a power supply module is also arranged in the housing, and the power supply module is electrically connected to the several lasers.

[0032] Furthermore, a switch control module is also arranged in the housing, and the switch control module is connected to the several lasers.

[0033] Compared with the prior art, in a virtual-real interaction device based on multiple infrared lasers of the present invention, the virtual-real interaction body can be used for a virtual-real interaction game scene based on a projector screen. An operator holds it, and the infrared lasers on it continuously emit laser light. After being projected onto the wall where the projector screen is located, the auxiliary camera captures the light spots, and the real-time pose of the prop body is obtained through an algorithm. This pose information can be used by the virtual-real interaction game to implement various interactive functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a structural diagram of a virtual-real interaction device provided in Embodiment 1;

[0035] Figure 2 is a structural diagram of the virtual-real interaction body provided in Embodiment 1;

[0036] Figure 3 is a flowchart of a pose calculation method of a virtual-real interaction device provided in Embodiment 2;

[0037] Figure 4 is a schematic diagram of laser emission provided in Embodiment 2;

[0038] Figure 5 is a schematic diagram of converting the image coordinates of the light spot into the coordinates in the projection plane coordinate system provided in Embodiment 2. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0040] The object of the present invention is to provide a virtual-real interaction-based device and its pose calculation method in view of the defects of the prior art.

[0041] Embodiment 1

[0042] This embodiment provides a virtual-real interaction-based device, as Figure 1 shown, including a virtual-real interaction body 1, a calculation unit 2, a camera 3, and a projection surface 4.

[0043] As Figure 2 shown, the virtual-real interaction body 1 includes a housing 11 and an inertial measurement unit IMU12, several infrared lasers 13, a power supply module (not shown in the figure), a switch control module (not shown in the figure), and a fixing structure arranged in the housing 11. Several infrared lasers 13 are all connected to the inertial measurement unit 12. In this embodiment, 4 infrared lasers 13 are taken as an example for illustration.

[0044] The housing 11 can have various shapes as long as it is convenient for operation. The fixing structure is a mechanical structure inside the housing for fixedly installing several infrared lasers to ensure their relative positions with the handle and other infrared lasers. This embodiment does not make specific limitations; at least 4 infrared lasers 13 are fixed on the housing 11, and the optical paths of the infrared lasers 13 are ensured to intersect at the same point by the fixing structure. It is necessary to ensure that the optical paths of the infrared lasers 13 do not have the situation of three lines being coplanar. The infrared lasers 13 are powered by a power supply module (battery) fixed inside the housing and can control the simultaneous on and off of the 4 infrared lasers 13 through a switch control module (such as a contact switch, etc.).

[0045] In this embodiment, each infrared laser emits a ray. If the rays of every two infrared lasers intersect at a point, then these two rays can form a definite plane. If the ray of the third infrared laser intersects with the previous two at the same point, it is necessary to ensure that this ray is not in the plane determined by the rays of the previous two, that is, a triangular pyramid can be formed, and thus it can be ensured that the optical paths of the infrared lasers do not have the situation of three lines being coplanar.

[0046] The projection surface 4 is a wall surface and / or a bottom surface. In this embodiment, the wall surface is taken as an example for illustration.

[0047] The camera 3 is arranged above the virtual-real interaction body 1. The camera 3 is communicatively connected to the computing unit 2. The camera 3 is used to photograph the wall surface. The light spot projected by the light emitted by the infrared laser 13 on the wall surface is captured by the camera 3 and sent to the computing module 2, and the computing module calculates the pose of the virtual-real interaction body 1.

[0048] The inertial measurement unit 12 is communicatively connected to the computing unit 2 through a wireless serial port module. The inertial measurement unit 12 sends acceleration data to the computing unit 2 in real time at a certain frequency. The computing unit 2 filters and interpolates the data to obtain an acceleration signal at equal time intervals, and further provides great convenience in processing for the subsequent usage scenarios of the prop through the signal at equal time intervals.

[0049] In this embodiment, the virtual-real interaction body can be used in a virtual-real interaction game scenario based on the projector screen. The operator holds it, and the infrared laser on it continuously emits laser light. After being projected onto the wall surface where the projector screen is located, the auxiliary camera photographs the light spot, and the real-time pose of the prop body is obtained through an algorithm. This pose information can be used by the virtual-real interaction game to implement various interactive functions.

[0050] Embodiment 2

[0051] This embodiment provides a method for calculating the pose based on a virtual-real interaction device, as Figure 3 shown, including:

[0052] S1. Obtain the coordinates of the projection of the laser emission in the preset virtual plane coordinate system to obtain the first coordinate;

[0053] S2. Obtain the light spot image of the laser emission collected by the camera from the emission point to the projection surface, and process the obtained light spot image to obtain the image coordinates of the light spot;

[0054] S3. Obtain the internal parameters and external parameters of the camera, and based on the internal parameters and external parameters of the camera, convert the image coordinates of the light spot into the coordinates in the projection surface coordinate system to obtain the second coordinate;

[0055] S4. According to the first coordinate and the second coordinate, and based on the EPnP algorithm, calculate the pose data of the virtual-real interaction device in the projection surface coordinate system.

[0056] In step S1, obtain the coordinates of the projection of the laser emission in the preset virtual plane coordinate system to obtain the first coordinate.

[0057] The relationship between the 4 infrared lasers is determined by a fixed structure. In this embodiment, one of the infrared lasers is taken as an example for illustration. First, a virtual-real interaction body coordinate system B is established. In the virtual-real interaction body coordinate system B, a virtual plane L is predefined B , and the plane coordinate system defined by the projection of the infrared laser on L B is , and its three-dimensional coordinates corresponding to B are , where z is L B The z-axis coordinate under B. This virtual projection point can be directly given by a fixed structure or conveniently measured using an auxiliary tool. In this embodiment, used to calculate in the normalized plane , and the latter is used to follow to solve the final pose together through the EPnP algorithm. is the coordinate of the image on the virtual plane in the prop body coordinate system; as Figure 4 shown, Figure 4 in any one of them.

[0058] In step S2, obtain the spot image of the laser beam emitted by the laser acquired by the camera, and process the acquired spot image to obtain the image coordinates of the spot.

[0059] The infrared laser emits laser light, which is emitted onto the wall. The camera acquires the image of the wall, and performs grayscale binary processing on the acquired image to find the white contours with areas within a preset threshold range in the grayscale binary processed image. When 4 contours are found, it is considered that all laser spots are found, and the centroid coordinates p i are calculated, that is, p i is the two-dimensional coordinate of the spot image.

[0060] In step S3, obtain the internal parameters and external parameters of the camera, and convert the image coordinates of the spot into the coordinates in the projection plane coordinate system based on the internal parameters and external parameters of the camera to obtain the second coordinate, as Figure 5 shown.

[0061] First, calibrate the camera to obtain the internal parameter K and external parameters of the camera. The internal parameter K includes focal length, principal distance, distortion coefficient, and adjacent pixel distance, etc. The external parameter is the position parameter of the camera relative to the wall coordinate system; where the camera coordinate system is defined as C, the wall coordinate system is defined as G, and the external parameter matrix of the camera to the wall coordinate system is defined as , and the external parameter matrix is a 4×4 homogeneous transformation matrix.

[0062] From the internal parameter K and external parameters of the camera, the coordinates of the spot in the wall coordinate system G are obtained, specifically:

[0063] S31. According to the image coordinates p i of the spot and the internal parameter K of the camera, calculate the direction vector of the spot formed by the laser beam emitted by the infrared laser in the camera coordinate system C, expressed as:

[0064] ;

[0065] Among them, represents the direction vector of the light spot in the camera coordinate system; K represents the internal parameters of the camera; p i represents the image coordinates of the light spot; C represents the camera coordinate system.

[0066] S32. Obtain the external parameter matrix of the camera , and calculate the rotation component and translation component of the external parameter matrix ;

[0067] The rotation component of the external parameter matrix is .

[0068] The translation component of the external parameter matrix is , which is expressed as:

[0069] ;

[0070] Among them, represents the translation component; G represents the wall coordinate system; t Cx represents the translation component of the x-axis; t Cy represents the translation component of the y-axis; t Cz represents the translation component of the z-axis; T represents the matrix transpose.

[0071] S33. Based on the rotation component convert the direction vector into the component in the projection plane coordinate system, which is expressed as:

[0072] ;

[0073] Among them, represents the component converted into the wall coordinate system, that is, a direction vector; represents the rotation component of the external parameter matrix; d ix represents the component on the x-axis; d iy represents the component on the y-axis; d iz represents the component on the z-axis; T represents the matrix transpose.

[0074] S34. According to the translation component and the converted component , calculate the three-dimensional physical coordinates of the image coordinates of the light spot converted into the wall coordinate system G to obtain the second coordinate, which is expressed as:

[0075] ;

[0076] Among them, It represents the second coordinate, that is, the image coordinates of the light spot are converted into the coordinates in the wall coordinate system.

[0077] In step S4, according to the first coordinate and the second coordinate , and based on the EPnP algorithm, calculate the transformation relationship between coordinate system G and coordinate system B , and then obtain the pose data of the virtual-real interaction ontology in the wall coordinate system G.

[0078] In this embodiment, the pose data is the pose matrix. For example, if the pose data is a 4X4 matrix, then the homogeneous matrix is rotated into a translation matrix, which is expressed as:

[0079]

[0080] Among them, is the rotation component, is the translation component.

[0081] In this embodiment, after step S4, it further includes:

[0082] S5. Post-process the pose data and transmit the post-processed pose data to the upper computer system.

[0083] In this embodiment, steps S1-S4 are all executed by the calculation unit, and after the calculation unit obtains the pose data, it will also perform post-processing steps such as filtering, interpolation, and prediction on the pose data to obtain pose signals at equal time intervals.

[0084] Due to the diversity of the corresponding relationship between the infrared laser and the wall light spot, multiple possible pose data solutions can be obtained from the above calculations. These solutions need to be combined with the acceleration signal obtained from the IMU to screen out the unique solution that conforms to the current IMU data. After post-processing, this unique solution is transmitted to the upper computer system for use through Ethernet or other means.

[0085] In this embodiment, the screening method can be selected according to the actual situation, and it is not limited to that proposed in this embodiment. The screening method is as follows: convert the rotation component of the obtained and the attitude data obtained from the IMU into quaternions. When the included angle between the direction vectors of the rotation axes of the two is considered to be the smallest, the corresponding is the unique solution that conforms to the current IMU data.

[0086] In this embodiment, the infrared laser continuously emits laser light. After being projected onto the wall where the projector screen is located, the auxiliary camera takes pictures of the light spot, and the real-time pose of the prop body is obtained through the algorithm. This pose information can be used by the virtual-real interaction game to realize various interactive functions.

[0087] Note that the above is only a preferred embodiment of the present invention and the applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A posture calculation method based on a virtual-reality interaction device, characterized in that: include: S1. Obtain the coordinates of the projection emitted by the laser in the preset virtual plane coordinate system to obtain the first coordinate; wherein the preset virtual plane coordinate system is a virtual plane predefined in the virtual-real interactive body coordinate system, and the plane coordinate system defined by the projection of the laser on the preset virtual plane coordinate system is the first coordinate; S2. Acquire the spot image of the laser emitted to the projection surface captured by the camera, and process the acquired spot image to obtain the image coordinates of the spot; S3. Obtaining the intrinsic parameters and extrinsic parameters of the camera, and converting the image coordinates of the light spot into coordinates in the projection surface coordinate system based on the intrinsic parameters and extrinsic parameters of the camera to obtain a second coordinate; S4. Calculate the position data of the virtual-real interaction device in the projection surface coordinate system based on the first coordinate and the second coordinate and based on the EPnP algorithm; The step S3 is specifically as follows: S31. Calculate the direction vector of the light spot in the camera coordinate system according to the image coordinates of the light spot and the intrinsic parameters of the camera; S32. Obtain the extrinsic matrix of the camera's extrinsic parameters, and calculate the rotational component and the translational component of the extrinsic matrix; S33. Converting the direction vector into a component in the projection surface coordinate system based on the rotation component; S34. According to the translation component and the component converted in step S33, the image coordinates of the calculated light spot are converted into coordinates in the projection surface coordinate system to obtain second coordinates.

2. A method for calculating a posture based on a virtual-reality interaction device according to claim 1, characterized in that: After step S4, the following steps are also included: S5. Post-process the posture data and transmit the posture data obtained by post-processing to the host computer system.

3. The method for calculating the position and posture based on the virtual-reality interaction device according to claim 1, characterized in that: The direction vector of the light spot in the camera coordinate system in step S31 is expressed as: in, represents the direction vector of the light spot in the camera coordinate system; K represents the intrinsic parameters of the camera; p i represents the image coordinates of the light spot; C represents the camera coordinate system.

4. A method for calculating a posture based on a virtual-reality interaction device according to claim 3, characterized in that: The translation component of the external parameter matrix in step S32 is expressed as: G t C =(t Cx ,t Cy ,t Cz ) T in, G t C represents the translation component; G represents the projection surface coordinate system; t Cx represents the translation component of the x-axis; t Cy represents the translation component of the y-axis; t Cz represents the translation component of the z-axis; T represents the matrix transpose.

5. A method for calculating a posture based on a virtual-reality interaction device according to claim 4, characterized in that: The components converted into the projection surface coordinate system in step S33 are expressed as: in, Represents the component converted to the projection surface coordinate system; G R C Represents the rotation component of the extrinsic matrix; d ix express The component on the x-axis; d iy express The component on the y-axis; d iz express Component along the z-axis; T denotes matrix transpose.

6. A method for calculating a posture based on a virtual-reality interaction device according to claim 5, characterized in that: The second coordinate in step S34 is expressed as: Among them, P i G It represents the second coordinate, that is, the image coordinate of the light spot is converted into the coordinate in the projection surface coordinate system.

7. A device based on virtual-reality interaction, characterized in that: It includes a virtual-reality interaction body, a computing unit, and a camera. The virtual-reality interaction body includes a shell and an inertial measurement unit and several lasers arranged in the shell. The several lasers are connected to the inertial measurement unit, and the computing units are connected to the inertial measurement unit and the camera. The computing unit is used to execute a posture calculation method based on a virtual-reality interaction device as described in any one of claims 1 to 6.

8. The device based on virtual-reality interaction according to claim 7, characterized in that: A power supply module is also arranged in the shell, and the power supply module is electrically connected to the plurality of lasers.

9. The device based on virtual-reality interaction according to claim 7, characterized in that: A switch control module is also arranged in the shell, and the switch control module is connected with a plurality of lasers.

Images