Video penetration mixed reality method, system, readable storage medium and electronic device

Abstract

The present invention discloses a video penetration mixed reality method, system, readable storage medium and electronic device, which are applied to mixed reality glasses. The method includes the following steps: obtaining real camera coordinates, and obtaining corresponding virtual camera coordinates, and obtaining mapping information of the two; obtaining a virtual world coordinate system, and obtaining a real world coordinate system according to the mapping information; for any object in the virtual world coordinate system, respectively obtaining the virtual coordinate point of the virtual world coordinate system and the real coordinate point in the real world coordinate system, and projecting the object in the virtual world coordinate system into the field of view of the mixed reality glasses, so that the object in the field of view is located at the real coordinate point. The present invention first obtains the virtual camera coordinates and the virtual world coordinate system according to the real camera coordinates, and then obtains the real world coordinate system with the help of the virtual world coordinate system, so that the object in the virtual coordinate system can be easily displayed in the real world coordinate system.

Description

Technical Field

[0001] The present invention relates to the field of mixed reality technology, and in particular to a video penetration mixed reality method, system, readable storage medium and electronic device. Background Art

[0002] Mixed Virtual Reality, or MR (Mixed Reality), is a combination of Virtual Reality VR (Virtual Reality) and Augmented Reality AR (Augmented Reality).

[0003] People can immerse themselves in a three-dimensional visual environment where the real world and virtual things are superimposed by computers through MR glasses terminals. Existing MR glasses can generate virtual things with strong real effects with the help of additional cameras with positioning functions.

[0004] Existing MR glasses must be used with additional cameras to realize the mixed reality function, and the implementation process is cumbersome. Summary of the invention

[0005] One object of the present invention is to provide a video penetration mixed reality method, system, readable storage medium and electronic device that are easier to implement.

[0006] A video penetration mixed reality method, applied to mixed reality glasses, comprises the following steps:

[0007] Acquire real camera coordinates, and acquire corresponding virtual camera coordinates, and acquire mapping information between the real camera coordinates and the virtual camera coordinates;

[0008] Acquire a virtual world coordinate system, and obtain a real world coordinate system according to the mapping information;

[0009] For any object in the virtual world coordinate system, obtain the virtual coordinate point of the virtual world coordinate system and the real coordinate point in the real world coordinate system respectively, and project the object in the virtual world coordinate system into the field of view of the mixed reality glasses so that the object in the field of view is located at the real coordinate point.

[0010] The beneficial effect of the present invention is: firstly obtain the virtual camera coordinates and the virtual world coordinate system according to the real camera coordinates, and then obtain the real world coordinate system with the help of the virtual world coordinate system, so that the objects in the virtual coordinate system can be easily displayed in the real world coordinate system.

[0011] In addition, the video penetration mixed reality method provided by the present invention may also have the following additional technical features:

[0012] Furthermore, the real environment world coordinate system and the virtual environment world coordinate system are both three-dimensional coordinates.

[0013] Furthermore, the mapping information includes the position and orientation of the camera.

[0014] Furthermore, the step of obtaining a real-world coordinate system according to the mapping information includes:

[0015] Acquire the virtual origin of the virtual world coordinate system, and obtain the real origin according to the mapping relationship;

[0016] Obtaining the length of the line between the virtual camera coordinates and the virtual origin, and the angles between the line and the virtual x-axis, virtual y-axis and virtual z-axis of the virtual world coordinate system, thereby obtaining the real x-axis, real y-axis and real z-axis;

[0017] The real world coordinate system is obtained by taking the real origin as a starting point and taking the real x-axis, real y-axis and real z-axis as coordinate axes.

[0018] Furthermore, the object in the virtual world coordinate system is a three-dimensional model, and the step of projecting the object into the field of view includes:

[0019] Obtaining the direction of the camera and the distance of the virtual camera from the virtual origin, thereby obtaining an image of the object captured from the virtual camera coordinates in real time;

[0020] The image is projected into the field of view.

[0021] Furthermore, the step of projecting the image into the field of view further includes:

[0022] Get the current time, and obtain the shadow length and azimuth according to the current time and the height of the object;

[0023] A shadow of the object is created with the shadow length and the azimuth angle.

[0024] Furthermore, the step of projecting the image into the field of view further includes:

[0025] Decomposing the image into a plurality of sub-image blocks;

[0026] Obtaining the distance between the virtual camera coordinates and the object, and determining whether the distance is greater than a preset value;

[0027] If so, when any obstacle is located between the real camera coordinates and the virtual object, an obstacle image is obtained, the obstacle image is superimposed in the field of view, and the sub-image block in the image that overlaps with the obstacle image is hidden.

[0028] Another object of the present invention is to provide a mixed reality system using the above-mentioned video penetration type, applied to mixed reality glasses, comprising:

[0029] A mapping acquisition module, used to acquire real camera coordinates, and acquire corresponding virtual camera coordinates, and acquire mapping information between the real camera coordinates and the virtual camera coordinates;

[0030] A coordinate system acquisition module, used to acquire a virtual world coordinate system and obtain a real world coordinate system according to the mapping information;

[0031] The projection module is used to obtain the virtual coordinate point of the virtual world coordinate system and the real coordinate point in the real world coordinate system for any object in the virtual world coordinate system, and project the object in the virtual world coordinate system into the field of view of the mixed reality glasses so that the object in the field of view is located at the real coordinate point.

[0032] The present invention also provides a readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned method is implemented.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0034] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0036] Figure 1 is a flowchart of a video penetration mixed reality method according to a first embodiment of the present invention;

[0037] Figure 2 is a schematic diagram of a video penetration mixed reality method according to a first embodiment of the present invention;

[0038] Figure 3 It is a structural block diagram of a video penetration mixed reality system according to a second embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. Several embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0040] See also Figure 1 , a first embodiment of the present invention proposes a video penetration mixed reality method, which is applied to mixed reality glasses and includes the following steps.

[0041] S1. Obtain real camera coordinates, obtain corresponding virtual camera coordinates, and obtain mapping information between the real camera coordinates and the virtual camera coordinates.

[0042] In this embodiment, the camera is disposed on the glasses, and no other additional cameras are required.

[0043] Specifically, the mapping information includes the position and orientation of the camera.

[0044] S2. Acquire a virtual world coordinate system, and obtain a real world coordinate system according to the mapping information.

[0045] In order to more accurately reflect the position of objects, the real environment world coordinate system and the virtual environment world coordinate system are both three-dimensional coordinates.

[0046] S3. For any object in the virtual world coordinate system, obtain the virtual coordinate point of the virtual world coordinate system and the real coordinate point in the real world coordinate system respectively, and project the object in the virtual world coordinate system into the field of view of the mixed reality glasses so that the object in the field of view is located at the real coordinate point.

[0047] Specifically, the step of obtaining a real-world coordinate system according to the mapping information includes:

[0048] S31. Obtain the virtual origin of the virtual world coordinate system, and obtain the real origin according to the mapping relationship;

[0049] S32. Obtain the length of the line between the virtual camera coordinates and the virtual origin, and the angles between the line and the virtual x-axis, virtual y-axis and virtual z-axis of the virtual world coordinate system, so as to obtain the real x-axis, real y-axis and real z-axis;

[0050] S33. Taking the real origin as the starting point and the real x-axis, real y-axis and real z-axis as coordinate axes, obtain the real world coordinate system.

[0051] Preferably, the object in the virtual world coordinate system is a three-dimensional model, and the step of projecting the object into the field of view includes:

[0052] S34. Obtaining the direction of the camera and the distance of the virtual camera from the virtual origin, so as to obtain the image of the object captured from the virtual camera coordinates in real time;

[0053] S35. Projecting the image into the field of view.

[0054] It can be understood that the direction of the camera can be understood as the viewing angle, and the position and size of the corresponding object in the field of view can be obtained according to the viewing angle and the angle between the connecting lines.

[0055] Furthermore, the step of projecting the image into the field of view further includes:

[0056] S351. Obtain the current time, and obtain the shadow length and azimuth according to the current time and the height of the object;

[0057] S352. Create the shadow of the object using the shadow length and the azimuth angle.

[0058] It should be noted that this function can be turned on when the user is outdoors.

[0059] In this embodiment, when the current time is 6-19 o'clock, the shadow length is obtained according to the sun altitude angle, and the shadow is not displayed at other times, making the virtual objects appear more realistic in the sun.

[0060] Furthermore, the step of projecting the image into the field of view further includes:

[0061] S353. Decomposing the image into a plurality of sub-image blocks;

[0062] S354. Obtain the distance between the virtual camera coordinates and the object, and determine whether the distance is greater than a preset value;

[0063] S355. If yes, when any obstacle is located between the real camera coordinates and the virtual object, obtain an obstacle image, superimpose the obstacle image in the field of view, and hide the sub-image block in the image that overlaps with the obstacle image.

[0064] In this embodiment, the preset value is 1m.

[0065] It should be pointed out that since virtual objects are projected in the field of view of the glasses, generally speaking, even if there are obstacles blocking part of the field of view, all virtual objects will still be displayed. For example, if a user extends his hand in front of the glasses to try to block part of the virtual objects, this cannot be done with traditional methods. The image of the virtual object will still cover the hand in the field of view, creating a sense of disharmony.

[0066] In this embodiment, since the image is divided into multiple sub-image blocks, the obstacle image in front is obtained through the camera, and part of the sub-image blocks are hidden according to the obstacle image, so that the virtual object can be partially blocked and looks more real.

[0067] The advantage of the present invention is that the virtual camera coordinates and the virtual world coordinate system are first obtained according to the real camera coordinates, and then the real world coordinate system is obtained with the help of the virtual world coordinate system, so that objects in the virtual coordinate system can be easily displayed in the real world coordinate system.

[0068] See also Figure 3 The second embodiment of the present invention proposes a video penetration mixed reality system using the above-mentioned mixed reality glasses, including:

[0069] A mapping acquisition module, used to acquire real camera coordinates, and acquire corresponding virtual camera coordinates, and acquire mapping information between the real camera coordinates and the virtual camera coordinates;

[0070] A coordinate system acquisition module, used to acquire a virtual world coordinate system and obtain a real world coordinate system according to the mapping information;

[0071] The projection module is used to obtain the virtual coordinate point of the virtual world coordinate system and the real coordinate point in the real world coordinate system for any object in the virtual world coordinate system, and project the object in the virtual world coordinate system into the field of view of the mixed reality glasses so that the object in the field of view is located at the real coordinate point.

[0072] This embodiment first obtains the virtual camera coordinates and the virtual world coordinate system according to the real camera coordinates, and then obtains the real world coordinate system with the help of the virtual world coordinate system, so that objects in the virtual coordinate system can be easily displayed in the real world coordinate system.

[0073] A third embodiment of the present invention provides a readable storage medium having computer instructions stored thereon, which implement the above-mentioned method when executed by a processor.

[0074] A fourth embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0075] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0076] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A video penetration mixed reality method, applied to mixed reality glasses, characterized in that: The following steps are involved: Acquire real camera coordinates, and acquire corresponding virtual camera coordinates, and acquire mapping information between the real camera coordinates and the virtual camera coordinates; Acquire a virtual world coordinate system, and obtain a real world coordinate system according to the mapping information; For any object in the virtual world coordinate system, respectively obtain a virtual coordinate point in the virtual world coordinate system and a real coordinate point in the real world coordinate system, and project the object in the virtual world coordinate system into the field of view of the mixed reality glasses so that the object in the field of view is located at the real coordinate point; The step of obtaining a real-world coordinate system according to the mapping information comprises: Acquire a virtual origin of the virtual world coordinate system, and obtain a real origin according to the mapping information; Obtaining the length of the line between the virtual camera coordinates and the virtual origin, and the angles between the line and the virtual x-axis, virtual y-axis and virtual z-axis of the virtual world coordinate system, thereby obtaining the real x-axis, real y-axis and real z-axis; The real world coordinate system is obtained by taking the real origin as a starting point and taking the real x-axis, real y-axis and real z-axis as coordinate axes.

2. The video penetration mixed reality method according to claim 1, characterized in that: The real world coordinate system and the virtual world coordinate system are both three-dimensional coordinates.

3. The video penetration mixed reality method according to claim 1, characterized in that: The mapping information includes the position and orientation of the camera.

4. The video penetration mixed reality method according to claim 1, characterized in that: The object in the virtual world coordinate system is a three-dimensional model, and the step of projecting the object into the field of view includes: Obtaining the direction of the camera and the distance of the virtual camera from the virtual origin, thereby obtaining an image of the object captured from the virtual camera coordinates in real time; The image is projected into the field of view.

5. The video penetration mixed reality method according to claim 4, characterized in that: The step of projecting the image into the field of view further comprises: Get the current time, and obtain the shadow length and azimuth according to the current time and the height of the object; A shadow of the object is created with the shadow length and the azimuth.

6. The video penetration mixed reality method according to claim 4, characterized in that: The step of projecting the image into the field of view further comprises: Decomposing the image into a plurality of sub-image blocks; Obtain the distance between the virtual camera coordinates and the object, and determine whether the distance is greater than a preset value; if so, when any obstacle is located between the real camera coordinates and the virtual object, obtain an obstacle image, superimpose the obstacle image in the field of view, and hide the sub-image block in the image that overlaps with the obstacle image.

7. A video penetration mixed reality system based on any of the methods of claims 1-6, applied to mixed reality glasses, characterized in that: include: A mapping acquisition module, used to acquire real camera coordinates, and acquire corresponding virtual camera coordinates, and acquire mapping information between the real camera coordinates and the virtual camera coordinates; A coordinate system acquisition module, used to acquire a virtual world coordinate system and obtain a real world coordinate system according to the mapping information; The projection module is used to obtain the virtual coordinate point of the virtual world coordinate system and the real coordinate point in the real world coordinate system for any object in the virtual world coordinate system, and project the object in the virtual world coordinate system into the field of view of the mixed reality glasses so that the object in the field of view is located at the real coordinate point.

8. A readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images