Naked eye 3D mixed reality interactive display system and method, electronic equipment and storage medium

Abstract

The invention discloses a naked-eye 3D mixed reality interaction display system and method, electronic equipment and a storage medium. The naked-eye 3D mixed reality interaction display system comprises a naked-eye 3D virtual reality interaction display system, a user interaction posture acquisition module, a virtual reality picture fusion module and a mixed reality rendering display module. The naked-eye 3D virtual reality interaction display system is used for displaying a naked-eye 3D virtual reality picture and supporting interaction with a virtual reality scene; the user interaction posture acquisition module is used for acquiring an interaction posture between a user and the virtual reality world in real time; the virtual reality picture fusion module is used for fusing a virtual screen picture and a virtual world interaction scene; and the mixed reality rendering display module is used for rendering and displaying a superimposed fusion picture of the virtual world interaction scene and the reality scene in real time. According to the naked-eye 3D mixed reality interactive display system and method, the electronic equipment and the storage medium provided by the invention, the sharing display capability of naked-eye 3D virtual reality interactive display can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of 3D display technology, and relates to a 3D display system, and in particular to a naked-eye 3D mixed reality interactive display system, method, electronic device and storage medium. Background Art

[0002] Virtual Reality (VR) technology creates a completely virtual scene by simulating human visual and auditory perception, allowing users to interact with the virtual environment through gestures, voice, head movements and other methods. This high degree of interactivity allows users to integrate into the virtual world more naturally and perform various operations and experiences, greatly enhancing the audience's sense of immersion. However, due to the limitations of current technology, VR technology still suffers from defects such as high technical barriers, dizziness and restraint when watching, etc. during its popularization.

[0003] Naked-eye 3D technology allows users to see the real 3D depth of field effect in real time without wearing any auxiliary equipment, which largely solves the sense of restraint and dizziness brought to users by traditional virtual reality devices such as VR glasses and VR helmets. At present, the virtual reality interaction system based on naked-eye 3D provides a more immersive interactive experience, allowing users to obtain higher-quality content and immersive feelings. However, the sharer cannot obtain this immersive interactive experience immediately, which greatly reduces the effectiveness of communication between the two parties in terms of technology, experience, etc., and also hinders the development of naked-eye 3D virtual reality interaction technology in different fields such as games, entertainment, education, medical care, industry, etc.

[0004] In view of this, there is an urgent need to design a new naked-eye 3D display method to overcome at least part of the above-mentioned defects of the existing naked-eye 3D display method. Summary of the invention

[0005] The present invention provides a naked-eye 3D mixed reality interactive display system, method, electronic device and storage medium, which can enhance the sharing and display capability of naked-eye 3D virtual reality interactive display.

[0006] To solve the above technical problem, according to one aspect of the present invention, the following technical solution is adopted:

[0007] A naked-eye 3D mixed reality interactive display system, the naked-eye 3D mixed reality interactive display system comprising:

[0008] Naked-eye 3D virtual reality interactive display system, used to display naked-eye 3D virtual reality images and support interaction with virtual reality scenes;

[0009] A user interaction posture acquisition module is used to acquire the interaction posture of the user and the virtual reality world in real time;

[0010] Virtual reality image fusion module, used to fuse virtual screen images and virtual world interactive scenes;

[0011] The mixed reality rendering and display module is used to render and display the superimposed fusion image of the virtual world interactive scene and the real scene in real time.

[0012] As an embodiment of the present invention, the naked-eye 3D virtual reality interactive display system includes a 3D virtual reality interactive system, a naked-eye 3D virtual reality display, an industrial camera module, an interactive light pen, and a computer system;

[0013] The industrial camera module includes a binocular human eye tracking camera and an interactive light pen positioning camera, and the industrial camera module obtains the position information of the binocular eyes and the interactive light pen;

[0014] The 3D virtual reality interaction system renders the 3D virtual reality interaction scene in 3D format, receives the interactive light pen position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D virtual reality display and the computer system;

[0015] The naked-eye 3D virtual reality display and the computer system are processed by an algorithm, and finally an interactive picture of a 3D virtual reality scene is displayed on the naked-eye 3D virtual reality display.

[0016] As an implementation mode of the present invention, the user interaction posture acquisition module includes a camera management module and an image processing and transmission module. The camera management module is used to acquire the interaction posture of the user and the naked-eye 3D virtual reality display terminal in real time. The image processing and transmission module is used to pre-process and transmit the acquired user interaction posture data.

[0017] The virtual reality screen fusion module includes a camera calibration module, a camera posture estimation module and an image fusion module. The camera calibration module is used to determine the spatial relationship between the camera and the calibration plane to obtain the camera's intrinsic parameter matrix. The camera posture estimation module is used to calculate the relative spatial position of the camera and the screen. The image fusion module is used to fuse the image data.

[0018] The mixed reality rendering and display module includes an image layered rendering and display module and an image overlay module. The image layered rendering and display module renders the screen image and the virtual interactive object respectively, realizing the layered rendering and display of the interactive object, the background image, and the screen. The image overlay module is to overlay and display the real image with the image of the virtual world to realize the mixed reality effect.

[0019] As an implementation mode of the present invention, the camera calibration module is used to perform camera calibration, extract feature points from the calibrated checkerboard image, and calculate the camera's intrinsic parameter matrix K by converting the camera coordinate system to the pixel coordinate system according to the camera pinhole model theory, which specifically includes:

[0020] Step A1: Set (u0, v0) as the projection coordinates of the camera optical axis in the pixel coordinate system. Then the transformation of a point p(x, y) in the image coordinate system to the pixel coordinate system can be expressed in homogeneous coordinate form:

[0021]

[0022] Among them, dx and dy represent the physical width and height of a pixel respectively;

[0023] Step A2: A point P(x c ,y c ,z c ) to the image coordinate system point p(x,y) is transformed into the homogeneous coordinates:

[0024]

[0025] Where f is the distance between the center of the camera coordinate system and the center of the image coordinate system, that is, the focal length;

[0026] Step A3: The transformation relationship from the camera coordinate system to the pixel coordinate system can be expressed as:

[0027]

[0028] Solving the correspondence between feature points can obtain the camera intrinsic parameter matrix K;

[0029]

[0030] The camera pose estimation module calculates the camera pose and uses a direct linear transformation (DLT) algorithm to obtain the transformation relationship between the world coordinate system and the camera coordinate system, and solves the transformation rotation matrix R and the translation matrix T, specifically including:

[0031] Step B1: The camera coordinate system matrix obtained by the camera calibration module can be used to obtain the conversion relationship between the world coordinate system and the camera coordinate system:

[0032]

[0033] From the above formula, we can know that two equations can be constructed for each feature point (3D-2D coordinate pair). When the number of feature points N≥6, a coordinate transformation equation group Ax=0 can be constructed, where A is a matrix composed of multiple groups of coordinate points. This equation group can be solved. Using the DLT algorithm, there are 12 unknowns in [R|T], x=[a1,…,a 12 ] T , the above formula can be changed into:

[0034]

[0035] Step B2: Use the least squares method to solve the equation system Ax=0 in step B1, and obtain the least squares solution under the constraint of |x|=1: argmin||Ax|| 2 , performing singular value decomposition (SVD) on A yields:

[0036] SVD(A)=USV T ;

[0037] Among them, A is an m×n matrix, U is an m×m orthogonal matrix, S is an m×n non-negative diagonal matrix, and V is an n×n orthogonal matrix;

[0038] The last column of the matrix V in step B3 and step B2 is the solution to the desired equation system Ax=0, because:

[0039] A = [KR KT];

[0040] Then the rotation matrix R and translation matrix T are obtained:

[0041]

[0042] During the virtual reality screen fusion calculation process, first obtain the virtual screen image position c and angle f;

[0043] From the camera pose calculation, we know that the virtual camera position C is: C = c*T;

[0044] The virtual camera angle F is: F = f*R;

[0045] The 3D virtual reality scene is displayed on the virtual screen, and the virtual world scene is interacted with through the virtual interactive light pen. The virtual camera fuses the virtual screen image and the virtual world interactive scene to obtain a virtual reality fusion image;

[0046] During the mixed reality rendering and display process, the mixed reality rendering and display module first renders the real picture captured by the camera on the bottom layer, then renders the background and scenes that do not affect the interaction in the 3D virtual reality interaction system into a virtual screen picture, and renders it on the upper layer of the screen in the real picture, and then renders the interactive 3D objects on the upper layer of the virtual screen picture, and masks the rest, and finally displays a mixed reality rendering picture with the 3D virtual reality interaction scene and the real scene superimposed.

[0047] According to another aspect of the present invention, the following technical solution is adopted: a naked-eye 3D mixed reality interactive display method, the naked-eye 3D mixed reality interactive display method comprising:

[0048] Naked-eye 3D virtual reality interactive display step: the naked-eye 3D virtual reality interactive display system displays the naked-eye 3D virtual reality picture and supports interaction with the virtual reality scene;

[0049] User interaction posture acquisition step: the user interaction posture acquisition module acquires the interaction posture of the user and the virtual reality world in real time;

[0050] Virtual reality screen fusion step: the virtual reality screen fusion module fuses the virtual screen image and the virtual world interactive scene;

[0051] Mixed reality rendering and display step: The mixed reality rendering and display module renders and displays the superimposed fusion picture of the virtual world interactive scene and the real scene in real time.

[0052] As an embodiment of the present invention, the naked-eye 3D virtual reality interactive display system includes a 3D virtual reality interactive system, a naked-eye 3D virtual reality display, an industrial camera module, an interactive light pen, and a computer system;

[0053] The industrial camera module includes a binocular human eye tracking camera and an interactive light pen positioning camera, and the industrial camera module obtains the position information of the binocular eyes and the interactive light pen;

[0054] The 3D virtual reality interaction system renders the 3D virtual reality interaction scene in 3D format, receives the interactive light pen position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D virtual reality display and the computer system;

[0055] The naked-eye 3D virtual reality display and the computer system are processed by an algorithm, and finally an interactive picture of a 3D virtual reality scene is displayed on the naked-eye 3D virtual reality display.

[0056] As an implementation mode of the present invention, the user interaction posture acquisition module includes a camera management module and an image processing and transmission module. The camera management module is used to acquire the interaction posture of the user and the naked-eye 3D virtual reality display terminal in real time. The image processing and transmission module is used to pre-process and transmit the acquired user interaction posture data.

[0057] The virtual reality screen fusion module includes a camera calibration module, a camera posture estimation module and an image fusion module. The camera calibration module is used to determine the spatial relationship between the camera and the calibration plane to obtain the camera's intrinsic parameter matrix. The camera posture estimation module is used to calculate the relative spatial position of the camera and the screen. The image fusion module is used to fuse the image data.

[0058] The mixed reality rendering and display module includes an image layered rendering and display module and an image overlay module. The image layered rendering and display module renders the screen image and the virtual interactive object respectively, realizing the layered rendering and display of the interactive object, the background image, and the screen. The image overlay module is to overlay and display the real image with the image of the virtual world to realize the mixed reality effect.

[0059] As an implementation mode of the present invention, the camera calibration module performs camera calibration, extracts feature points from the calibrated checkerboard image, and calculates the camera's intrinsic parameter matrix K by transforming the camera coordinate system to the pixel coordinate system according to the camera pinhole model theory, specifically including:

[0060] Step A1: Set (u0, v0) as the projection coordinates of the camera optical axis in the pixel coordinate system. Then the transformation of a point p(x, y) in the image coordinate system to the pixel coordinate system can be expressed in homogeneous coordinate form:

[0061]

[0062] Among them, dx and dy represent the physical width and height of a pixel respectively;

[0063] Step A2: A point P(x c ,y c ,z c ) to the image coordinate system point p(x,y) is transformed into the homogeneous coordinates:

[0064]

[0065] Where f is the distance between the center of the camera coordinate system and the center of the image coordinate system, that is, the focal length;

[0066] Step A3: The transformation relationship from the camera coordinate system to the pixel coordinate system can be expressed as:

[0067]

[0068] Solving the correspondence between feature points can obtain the camera intrinsic parameter matrix K;

[0069]

[0070] The camera pose estimation module calculates the camera pose and uses a direct linear transformation (DLT) algorithm to obtain the transformation relationship between the world coordinate system and the camera coordinate system, and solves the transformation rotation matrix R and the translation matrix T, specifically including:

[0071] Step B1: The camera coordinate system matrix obtained by the camera calibration module can be used to obtain the conversion relationship between the world coordinate system and the camera coordinate system:

[0072]

[0073] From the above formula, we can know that two equations can be constructed for each feature point (3D-2D coordinate pair). When the number of feature points N≥6, a coordinate transformation equation group Ax=0 can be constructed, where A is a matrix composed of multiple groups of coordinate points. This equation group can be solved. Using the DLT algorithm, there are 12 unknowns in [R|T], x=[a1,…,a 12 ] T , the above formula can be changed into:

[0074]

[0075] Step B2: Use the least squares method to solve the equation system Ax=0 in step B1, and obtain the least squares solution under the constraint of |x|=1: argmin||Ax|| 2 , performing singular value decomposition (SVD) on A yields:

[0076] SVD(A)=VSV T ;

[0077] Among them, A is an m×n matrix, U is an m×m orthogonal matrix, S is an m×n non-negative diagonal matrix, and V is an n×n orthogonal matrix;

[0078] The last column of the matrix V in step B3 and step B2 is the solution to the desired equation system Ax=0, because:

[0079] A = [KR KT];

[0080] Then the rotation matrix R and translation matrix T are obtained:

[0081]

[0082]

[0083] During the virtual reality screen fusion calculation process, first obtain the virtual screen image position c and angle f;

[0084] From the camera pose calculation, we know that the virtual camera position C is: C = c*T;

[0085] The virtual camera angle F is: F = f*R;

[0086] The 3D virtual reality scene is displayed on the virtual screen, and the virtual world scene is interacted with through the virtual interactive light pen. The virtual camera fuses the virtual screen image and the virtual world interactive scene to obtain a virtual reality fusion image;

[0087] During the mixed reality rendering and display process, the mixed reality rendering and display module first renders the real picture captured by the camera on the bottom layer, then renders the background and scenes that do not affect the interaction in the 3D virtual reality interaction system into a virtual screen picture, and renders it on the upper layer of the screen in the real picture, and then renders the interactive 3D objects on the upper layer of the virtual screen picture, and masks the rest, and finally displays a mixed reality rendering picture with the 3D virtual reality interaction scene and the real scene superimposed.

[0088] According to another aspect of the present invention, the following technical solution is adopted: an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0089] According to another aspect of the present invention, the following technical solution is adopted: a storage medium stores computer program instructions, and the computer program instructions implement the steps of the above method when executed by a processor.

[0090] The beneficial effects of the present invention are as follows: the naked-eye 3D mixed reality interactive display system, method, electronic device and storage medium proposed in the present invention can enhance the sharing and display capabilities of naked-eye 3D virtual reality interactive display.

[0091] As a fusion of mixed reality and naked-eye 3D technology, the present invention not only inherits the advantages of both, but also realizes the seamless integration of the naked-eye 3D virtual world and the real world through innovative technical means, providing users with a richer, more intuitive and realistic experience. At the same time, the realistic and immersive naked-eye 3D effects viewed by users are shared, thus building an immersive interactive naked-eye 3D mixed reality platform for the dissemination and exchange of naked-eye 3D virtual reality content. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1Schematic diagram of the composition of a naked-eye 3D mixed reality interactive display system in one embodiment of the present invention.

[0093] Figure 2 Flow chart of a naked-eye 3D mixed reality interactive display method in one embodiment of the present invention.

[0094] Figure 3 Schematic diagram of the composition of a naked-eye 3D mixed reality interactive display system in one embodiment of the present invention.

[0095] Figure 4 Schematic diagram of the composition of a naked-eye 3D virtual reality interactive display system in one embodiment of the present invention.

[0096] Figure 5 This is a flow chart of naked-eye 3D virtual reality interactive display of the present invention.

[0097] Figure 6 Schematic diagram of the relationship between coordinate systems in camera calibration and pose calculation of the present invention.

[0098] Figure 7 This is a flow chart of virtual reality screen fusion of the present invention.

[0099] Figure 8 This is a flow chart of the mixed reality rendering display algorithm of the present invention.

[0100] Fig. 9 FIG. 4 is a schematic diagram of the composition of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0101] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0102] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0103] The description in this section is only for several typical embodiments, and the present invention is not limited to the scope of the embodiments. The same or similar prior art means and some technical features in the embodiments are mutually replaced within the scope of the present invention.

[0104] The description of the steps in each embodiment in the specification is only for the convenience of explanation, and the implementation method of the present application is not limited by the order of implementation of the steps.

[0105] The term "connection" as used in the specification includes both direct connection and indirect connection, such as connection through some active devices, passive devices or electrically conductive media; it may also include connection through other active devices or passive devices known to those skilled in the art on the basis of achieving the same or similar functional purposes, such as connection through circuits or components such as switches and follower circuits.

[0106] The present invention discloses a naked eye 3D mixed reality interactive display system. Figure 1 This is a schematic diagram of the composition of a naked-eye 3D mixed reality interactive display system in one embodiment of the present invention; please refer to Figure 1 The naked-eye 3D mixed reality interactive display system includes: a naked-eye 3D virtual reality interactive display system 1, a user interaction posture acquisition module 2, a virtual reality picture fusion module 3 and a mixed reality rendering display module 4.

[0107] The naked-eye 3D virtual reality interactive display system 1 is used to display naked-eye 3D virtual reality images and support interaction with virtual reality scenes; the user interaction posture acquisition module 2 is used to acquire the interaction posture of the user and the virtual reality world in real time; the virtual reality picture fusion module 3 is used to fuse the virtual screen picture and the virtual world interaction scene; the mixed reality rendering and display module 4 is used to render and display the superimposed fusion picture of the virtual world interaction scene and the real scene in real time.

[0108] Figure 3 Schematic diagram of the composition of a naked-eye 3D mixed reality interactive display system in an embodiment of the present invention, which includes a naked-eye 3D virtual reality display terminal 10, an interactive light pen 20, a camera 30, and a common display device 40. The user watches the naked-eye 3D virtual reality content in real time through the naked-eye 3D virtual reality terminal 10, and interacts with the naked-eye 3D virtual reality world through the interactive light pen 20. The camera 30 obtains the user's interactive posture in real time, and displays the user's interactive screen with the naked-eye 3D virtual reality on the common display device 40 in real time in the form of mixed reality.

[0109] The naked eye 3D virtual reality interactive display system 1 is used to render naked eye 3D effects in real time and interact with virtual reality scenes. Figure 3 As shown, in one embodiment of the present invention, the naked eye 3D virtual reality interactive display system 1 includes an industrial camera module 15, a naked eye 3D display screen 16, an interactive light pen 17 and a computer system. Figure 5As shown, the industrial camera module 15 obtains the spatial position coordinates of both eyes and the spatial position and posture of the interactive light pen in real time, and transmits signals to the naked-eye 3D virtual reality interactive display system 1. The naked-eye 3D virtual reality interactive display system 1 renders the 3D virtual reality content in real time according to the position of the interactive light pen 17 and outputs it in 3D format, and transmits the image signal to the naked-eye 3D display screen 16 and the computer system. The naked-eye 3D virtual reality display system 1 and the computer system are processed by algorithms and finally display the interactive screen of the real-time naked-eye 3D virtual reality scene.

[0110] The user interaction posture acquisition module 2 is used to acquire the interaction posture of the user and the virtual world in real time, and to process and transmit the data. In one embodiment of the present invention, the user interaction posture acquisition module 2 detects the working state of the camera through the camera management module, and after the camera is normally started, adjusts the camera position and places it in a suitable shooting position, acquires the interaction posture of the user and the naked-eye 3D virtual reality display terminal through shooting, and transmits the data to the image processing and transmission module for processing through the USB transmission interface.

[0111] The virtual reality picture fusion module 3 is used to fuse the pictures of the virtual scene and the virtual display plane interaction. In one embodiment of the present invention, the composition of the virtual reality picture fusion module 3 is as follows: Figure 7 As shown, it includes a camera calibration module, a camera pose estimation module and an image fusion module. The camera calibration module is used to determine the internal parameters of the camera. The camera pose estimation module is used to inversely calculate the relative spatial position and pose of the camera and the screen. The image fusion module refers to the fusion processing of image data. Specifically, the process of camera calibration and camera pose calculation mainly involves the conversion between four coordinate systems; Figure 6 As shown, there are pixel coordinate system 8, image coordinate system 9, camera coordinate system 10 and world coordinate system 11 respectively.

[0112] The camera calibration module is used to perform camera calibration, extract feature points from the calibrated checkerboard image, and calculate the camera's intrinsic parameter matrix K by converting the camera coordinate system to the pixel coordinate system according to the camera pinhole model theory. Specifically:

[0113] ① Set (u0, v0) as the projection coordinates of the camera optical axis in the pixel coordinate system, then the transformation of a point p(x, y) in the image coordinate system to the pixel coordinate system can be expressed in homogeneous coordinate form:

[0114]

[0115] Where dx and dy represent the physical width and height of a pixel respectively.

[0116] ② A point P(x c ,y c ,zc ) to the image coordinate system point p(x,y) is transformed into the homogeneous coordinates:

[0117]

[0118] Where f is the distance between the center of the camera coordinate system and the center of the image coordinate system, that is, the focal length.

[0119] ③Then the transformation relationship from the camera coordinate system to the pixel coordinate system can be expressed by ① and ②:

[0120]

[0121] The camera intrinsic parameter matrix K can be obtained by solving the corresponding relationship of the feature points.

[0122]

[0123] The camera pose estimation module calculates the camera pose and uses the Direct Linear Transform (DLT) algorithm to obtain the transformation relationship between the world coordinate system and the camera coordinate system, and solves the transformation rotation matrix R and the translation matrix T. Specifically:

[0124] ① The camera coordinate system matrix obtained by the camera calibration module can be used to obtain the conversion relationship between the world coordinate system and the camera coordinate system:

[0125]

[0126] From the above formula, we can know that two equations can be constructed for each feature point (3D-2D coordinate pair). When the number of feature points N≥6, a coordinate transformation equation group Ax=0 can be constructed, where A is a matrix composed of multiple sets of coordinate points. This equation group can be solved. Using the DLT algorithm, there are 12 unknowns in [R|T], x=[a1,…,a 12 ] T , the above formula can be changed into:

[0127]

[0128] ②Use the least squares method to solve the equation system Ax=0 in ①, and you can get the least squares solution under the constraint of |x|=1: argmin||Ax|| 2 , performing singular value decomposition (SVD) on A yields,

[0129] SVD(A)=USV T

[0130] Where A is an m×n matrix, U is an m×m orthogonal matrix, S is an m×n non-negative diagonal matrix, and V is an n×n orthogonal matrix.

[0131] ③The last column of the matrix V in step ② is the solution to the desired equation system Ax=0, because:

[0132] A=[KR KT]

[0133] Then the rotation matrix R and translation matrix T are obtained:

[0134]

[0135] During the virtual reality screen fusion calculation process, the virtual screen position c and angle f are first obtained. From the camera pose calculation, it can be known that the virtual camera position C is:

[0136] C=c*T,

[0137] The virtual camera angle F is:

[0138] F=f*R,

[0139] The 3D virtual reality scene is displayed on the virtual screen, and the virtual world scene is interacted with through the virtual interactive light pen. The virtual camera fuses the virtual screen image and the virtual world interactive scene to obtain a virtual reality fusion image.

[0140] The mixed reality rendering and display module 4 is used to render the virtual world interactive scene and the real scene in real time, and display it on a normal display. Figure 8 As shown, the mixed reality rendering display module 4 first renders the real picture taken by the camera on the bottom layer, and then renders the background and scenes that do not affect the interaction in the 3D virtual reality interaction system into a virtual screen picture, and renders it on the upper layer of the screen position in the real picture, and then renders the 3D objects that can be interacted with or the objects that will affect the interaction on the upper layer of the virtual screen picture, and the rest are masked, and finally a mixed reality rendering picture with the 3D virtual reality interaction scene and the real scene superimposed is displayed.

[0141] The present invention further discloses a naked-eye 3D mixed reality interactive display method, the naked-eye 3D mixed reality interactive display method comprising:

[0142] [Step S1] Naked-eye 3D virtual reality interactive display step: the naked-eye 3D virtual reality interactive display system displays the naked-eye 3D virtual reality picture and supports interaction with the virtual reality scene;

[0143] [Step S2] User interaction posture acquisition step: The user interaction posture acquisition module acquires the interaction posture of the user and the virtual reality world in real time;

[0144] [Step S3] Virtual reality picture fusion step: the virtual reality picture fusion module fuses the virtual screen picture and the virtual world interactive scene;

[0145] [Step S4] Mixed reality rendering and display step: The mixed reality rendering and display module renders and displays the superimposed fusion picture of the virtual world interactive scene and the real scene in real time.

[0146] In one embodiment of the present invention, the naked-eye 3D virtual reality interactive display system comprises a 3D virtual reality interactive system, a naked-eye 3D virtual reality display, an industrial camera module, an interactive light pen, and a computer system;

[0147] The industrial camera module includes a binocular human eye tracking camera and an interactive light pen positioning camera, and the industrial camera module obtains the position information of the binocular eyes and the interactive light pen;

[0148] The 3D virtual reality interaction system renders the 3D virtual reality interaction scene in 3D format, receives the interactive light pen position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D virtual reality display and the computer system;

[0149] The naked-eye 3D virtual reality display and the computer system are processed by an algorithm, and finally an interactive picture of a 3D virtual reality scene is displayed on the naked-eye 3D virtual reality display.

[0150] The user interaction posture acquisition module includes a camera management module and an image processing and transmission module. The camera management module is used to acquire the interaction posture of the user and the naked-eye 3D virtual reality display terminal in real time. The image processing and transmission module is used to pre-process and transmit the acquired user interaction posture data.

[0151] The virtual reality screen fusion module includes a camera calibration module, a camera posture estimation module and an image fusion module. The camera calibration module is used to determine the spatial relationship between the camera and the calibration plane to obtain the camera's intrinsic parameter matrix. The camera posture estimation module is used to calculate the relative spatial position of the camera and the screen. The image fusion module is used to fuse the image data.

[0152] The mixed reality rendering and display module includes an image layered rendering and display module and an image overlay module. The image layered rendering and display module renders the screen image and the virtual interactive object respectively, realizing the layered rendering and display of the interactive object, the background image, and the screen. The image overlay module is to overlay and display the real image with the image of the virtual world to realize the mixed reality effect.

[0153] In one use scenario of the present invention, the camera calibration module performs camera calibration, extracts feature points from the calibrated checkerboard image, and calculates the camera's intrinsic parameter matrix K by transforming the camera coordinate system into the pixel coordinate system according to the camera pinhole model theory, specifically including:

[0154] Step A1: Set (u0, v0) as the projection coordinates of the camera optical axis in the pixel coordinate system. Then the transformation of a point p(x, y) in the image coordinate system to the pixel coordinate system can be expressed in homogeneous coordinate form:

[0155]

[0156] Among them, dx and dy represent the physical width and height of a pixel respectively;

[0157] Step A2: A point P(x c ,y c ,z c ) to the image coordinate system point p(x,y) is transformed into the homogeneous coordinates:

[0158]

[0159] Where f is the distance between the center of the camera coordinate system and the center of the image coordinate system, that is, the focal length;

[0160] Step A3: The transformation relationship from the camera coordinate system to the pixel coordinate system can be expressed as:

[0161]

[0162] Solving the correspondence between feature points can obtain the camera intrinsic parameter matrix K;

[0163]

[0164] The camera pose estimation module calculates the camera pose and uses a direct linear transformation (DLT) algorithm to obtain the transformation relationship between the world coordinate system and the camera coordinate system, and solves the transformation rotation matrix R and the translation matrix T, specifically including:

[0165] Step B1: The camera coordinate system matrix obtained by the camera calibration module can be used to obtain the conversion relationship between the world coordinate system and the camera coordinate system:

[0166]

[0167] From the above formula, we can know that two equations can be constructed for each feature point (3D-2D coordinate pair). When the number of feature points N≥6, a coordinate transformation equation group Ax=0 can be constructed, where A is a matrix composed of multiple groups of coordinate points. This equation group can be solved. Using the DLT algorithm, there are 12 unknowns in [R|T], x=[a1,…,a 12 ] T , the above formula can be changed into:

[0168]

[0169] Step B2: Use the least squares method to solve the equation system Ax=0 in step B1, and obtain the least squares solution under the constraint of |x|=1: argmin||Ax|| 2 , performing singular value decomposition (SVD) on A yields:

[0170] SVD(A)=USV T ;

[0171] Among them, A is an m×n matrix, U is an m×m orthogonal matrix, S is an m×n non-negative diagonal matrix, and V is an n×n orthogonal matrix;

[0172] The last column of the matrix V in step B3 and step B2 is the solution to the desired equation system Ax=0, because:

[0173] A = [KRKT];

[0174] Then the rotation matrix R and translation matrix T are obtained:

[0175]

[0176] During the virtual reality screen fusion calculation process, first obtain the virtual screen image position c and angle f;

[0177] From the camera pose calculation, we know that the virtual camera position C is: C = c*T;

[0178] The virtual camera angle F is: F = f*R;

[0179] The 3D virtual reality scene is displayed on the virtual screen, and the virtual world scene is interacted with through the virtual interactive light pen. The virtual camera fuses the virtual screen image and the virtual world interactive scene to obtain a virtual reality fusion image;

[0180] During the mixed reality rendering and display process, the mixed reality rendering and display module first renders the real picture captured by the camera on the bottom layer, then renders the background and scenes that do not affect the interaction in the 3D virtual reality interaction system into a virtual screen picture, and renders it on the upper layer of the screen in the real picture, and then renders the interactive 3D objects on the upper layer of the virtual screen picture, and masks the rest, and finally displays a mixed reality rendering picture with the 3D virtual reality interaction scene and the real scene superimposed.

[0181] The present invention also discloses an electronic device, Fig. 9 FIG. 1 is a schematic diagram of the composition of an electronic device in an embodiment of the present invention; please refer to FIG. Fig. 9At the hardware level, the electronic device includes a memory, a processor and at least one network interface; the processor may be a microprocessor, and the memory may include a memory, such as a random access memory (RAM), or a non-volatile memory, etc. Of course, the electronic device may also be provided with other hardware as required.

[0182] The processor, network interface and memory may be interconnected via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus may include an address bus, a data bus, a control bus, etc. The memory is used to store programs (which may include operating system programs and application programs); the program may include program code, which may include computer operation instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0183] In one embodiment, the processor can read the corresponding program from the non-volatile memory into the memory and then run it; the processor can execute the program stored in the memory and is specifically used to perform the following operations (such as Figure 2 shown):

[0184] [Step S1] Naked-eye 3D virtual reality interactive display step: the naked-eye 3D virtual reality interactive display system displays the naked-eye 3D virtual reality picture and supports interaction with the virtual reality scene;

[0185] [Step S2] User interaction posture acquisition step: The user interaction posture acquisition module acquires the interaction posture of the user and the virtual reality world in real time;

[0186] [Step S3] Virtual reality picture fusion step: the virtual reality picture fusion module fuses the virtual screen picture and the virtual world interactive scene;

[0187] [Step S4] Mixed reality rendering and display step: The mixed reality rendering and display module renders and displays the superimposed fusion picture of the virtual world interactive scene and the real scene in real time.

[0188] The present invention further discloses a storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the following steps of the method of the present invention (eg Figure 2 shown):

[0189] [Step S1] Naked-eye 3D virtual reality interactive display step: the naked-eye 3D virtual reality interactive display system displays the naked-eye 3D virtual reality picture and supports interaction with the virtual reality scene;

[0190] [Step S2] User interaction posture acquisition step: The user interaction posture acquisition module acquires the interaction posture of the user and the virtual reality world in real time;

[0191] [Step S3] Virtual reality picture fusion step: the virtual reality picture fusion module fuses the virtual screen picture and the virtual world interactive scene;

[0192] [Step S4] Mixed reality rendering and display step: The mixed reality rendering and display module renders and displays the superimposed fusion picture of the virtual world interactive scene and the real scene in real time.

[0193] In summary, the naked-eye 3D mixed reality interactive display system, method, electronic device and storage medium proposed in the present invention can enhance the sharing and display capabilities of naked-eye 3D virtual reality interactive display.

[0194] As a fusion of mixed reality and naked-eye 3D technology, the present invention not only inherits the advantages of both, but also realizes the seamless integration of the naked-eye 3D virtual world and the real world through innovative technical means, providing users with a richer, more intuitive and realistic experience. At the same time, the realistic and immersive naked-eye 3D effects viewed by users are shared, thus building an immersive interactive naked-eye 3D mixed reality platform for the dissemination and exchange of naked-eye 3D virtual reality content.

[0195] It should be noted that the present application can be implemented in software and / or a combination of software and hardware; for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium; for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present application can be implemented in hardware; for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0196] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0197] The description and application of the present invention here are illustrative, and it is not intended to limit the scope of the present invention to the above-mentioned embodiments. The effects or advantages involved in the embodiments may not be embodied in the embodiments due to interference from various factors, and the description of the effects or advantages is not used to limit the embodiments. The deformation and change of the embodiments disclosed here are possible, and the replacement of the embodiments and the various equivalent parts are well known to those of ordinary skill in the art. It should be clear to those skilled in the art that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential features of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.

Claims

1. A naked-eye 3D mixed reality interactive display system, characterized in that: The naked eye 3D mixed reality interactive display system comprises: Naked-eye 3D virtual reality interactive display system, used to display naked-eye 3D virtual reality images and support interaction with virtual reality scenes; A user interaction posture acquisition module is used to acquire the interaction posture of the user and the virtual reality world in real time; Virtual reality image fusion module, used to fuse virtual screen images and virtual world interactive scenes; The mixed reality rendering and display module is used to render and display the superimposed fusion image of the virtual world interactive scene and the real scene in real time.

2. The naked eye 3D mixed reality interactive display system according to claim 1, characterized in that: The naked-eye 3D virtual reality interactive display system comprises a 3D virtual reality interactive system, a naked-eye 3D virtual reality display, an industrial camera module, an interactive light pen and a computer system; The industrial camera module includes a binocular human eye tracking camera and an interactive light pen positioning camera, and the industrial camera module obtains the position information of the binocular eyes and the interactive light pen; The 3D virtual reality interaction system renders the 3D virtual reality interaction scene in 3D format, receives the interactive light pen position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D virtual reality display and the computer system; The naked-eye 3D virtual reality display and the computer system are processed by an algorithm, and finally an interactive picture of a 3D virtual reality scene is displayed on the naked-eye 3D virtual reality display.

3. The naked eye 3D mixed reality interactive display system according to claim 1, characterized in that: The user interaction posture acquisition module includes a camera management module and an image processing and transmission module. The camera management module is used to acquire the interaction posture of the user and the naked-eye 3D virtual reality display terminal in real time. The image processing and transmission module is used to pre-process and transmit the acquired user interaction posture data. The virtual reality screen fusion module includes a camera calibration module, a camera posture estimation module and an image fusion module. The camera calibration module is used to determine the spatial relationship between the camera and the calibration plane to obtain the camera's intrinsic parameter matrix. The camera posture estimation module is used to calculate the relative spatial position of the camera and the screen. The image fusion module is used to fuse the image data. The mixed reality rendering and display module includes an image layered rendering and display module and an image overlay module. The image layered rendering and display module renders the screen image and the virtual interactive object respectively, realizing the layered rendering and display of the interactive object, the background image, and the screen. The image overlay module is to overlay and display the real image with the image of the virtual world to realize the mixed reality effect.

4. The naked eye 3D mixed reality interactive display method according to claim 3, characterized in that: The camera calibration module extracts feature points from a set checkerboard image. According to the camera pinhole model theory, the camera's intrinsic parameter matrix K can be calculated by transforming the camera coordinate system to the pixel coordinate system, which specifically includes: Step A1: Set (u0, v0) as the projection coordinates of the camera optical axis in the pixel coordinate system. Then the transformation of a point p(x, y) in the image coordinate system to the pixel coordinate system can be expressed in homogeneous coordinate form: Among them, dx and dy represent the physical width and height of a pixel respectively; Step A2: A point P(x c ,y c ,z c ) to the image coordinate system point p(x,y) is transformed into the homogeneous coordinates: Where f is the distance between the center of the camera coordinate system and the center of the image coordinate system, that is, the focal length; Step A3: The transformation relationship from the camera coordinate system to the pixel coordinate system can be expressed as: Solving the correspondence between feature points can obtain the camera intrinsic parameter matrix K; The camera pose estimation module calculates the camera pose and uses a direct linear transformation algorithm to obtain the transformation relationship between the world coordinate system and the camera coordinate system, and solves the transformation rotation matrix R and the translation matrix T, specifically including: Step B1: The camera coordinate system matrix obtained by the camera calibration module can be used to obtain the conversion relationship between the world coordinate system and the camera coordinate system: From the above formula, we can know that two equations can be constructed for each feature point. When the number of feature points N≥6, a coordinate transformation equation group Ax=0 can be constructed, where A is a matrix composed of multiple sets of coordinate points. This equation group can be solved. Using the DLT algorithm, there are 12 unknowns in [R|T], x=[a1,…,a 12 ] T , the above formula can be changed into: Step B2: Use the least squares method to solve the equation system Ax=0 in step B1, and obtain the least squares solution under the constraint of |x|=1: argmin||Ax|| 2 , performing singular value decomposition on A yields: SVD(A)=USV T ; Among them, A is an m×n matrix, U is an m×m orthogonal matrix, S is an m×n non-negative diagonal matrix, and V is an n×n orthogonal matrix; The last column of the matrix V in step B3 and step B2 is the solution to the desired equation system Ax=0, because: A = [KR KT]; Then the rotation matrix R and translation matrix T are obtained: During the virtual reality screen fusion calculation process, first obtain the virtual screen image position c and angle f; From the camera pose calculation, we know that the virtual camera position C is: C = c*T; The virtual camera angle F is: F = f*R; The 3D virtual reality scene is displayed on the virtual screen, and the virtual world scene is interacted with through the virtual interactive light pen. The virtual camera fuses the virtual screen image and the virtual world interactive scene to obtain a virtual reality fusion image; During the mixed reality rendering and display process, the mixed reality rendering and display module first renders the real picture captured by the camera on the bottom layer, then renders the background and scenes that do not affect the interaction in the 3D virtual reality interaction system into a virtual screen picture, and renders it on the upper layer of the screen in the real picture, and then renders the interactive 3D objects on the upper layer of the virtual screen picture, and masks the rest, and finally displays a mixed reality rendering picture with the 3D virtual reality interaction scene and the real scene superimposed.

5. A naked eye 3D mixed reality interactive display method, characterized in that: The naked eye 3D mixed reality interactive display method comprises: Naked-eye 3D virtual reality interactive display step: the naked-eye 3D virtual reality interactive display system displays the naked-eye 3D virtual reality picture and supports interaction with the virtual reality scene; User interaction posture acquisition step: the user interaction posture acquisition module acquires the interaction posture of the user and the virtual reality world in real time; Virtual reality screen fusion step: the virtual reality screen fusion module fuses the virtual screen image and the virtual world interactive scene; Mixed reality rendering and display step: The mixed reality rendering and display module renders and displays the superimposed fusion picture of the virtual world interactive scene and the real scene in real time.

6. The naked eye 3D mixed reality interactive display method according to claim 5, characterized in that: The naked-eye 3D virtual reality interactive display system comprises a 3D virtual reality interactive system, a naked-eye 3D virtual reality display, an industrial camera module, an interactive light pen and a computer system; The industrial camera module includes a binocular human eye tracking camera and an interactive light pen positioning camera, and the industrial camera module obtains the position information of the binocular eyes and the interactive light pen; The 3D virtual reality interaction system renders the 3D virtual reality interaction scene in 3D format, receives the interactive light pen position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D virtual reality display and the computer system; The naked-eye 3D virtual reality display and the computer system are processed by an algorithm, and finally an interactive picture of a 3D virtual reality scene is displayed on the naked-eye 3D virtual reality display.

7. The naked eye 3D mixed reality interactive display method according to claim 5, characterized in that: The user interaction posture acquisition module includes a camera management module and an image processing and transmission module. The camera management module is used to acquire the interaction posture of the user and the naked-eye 3D virtual reality display terminal in real time. The image processing and transmission module is used to pre-process and transmit the acquired user interaction posture data. The virtual reality screen fusion module includes a camera calibration module, a camera posture estimation module and an image fusion module. The camera calibration module is used to determine the spatial relationship between the camera and the calibration plane to obtain the camera's intrinsic parameter matrix. The camera posture estimation module is used to calculate the relative spatial position of the camera and the screen. The image fusion module is used to fuse the image data. The mixed reality rendering and display module includes an image layered rendering and display module and an image overlay module. The image layered rendering and display module renders the screen image and the virtual interactive object respectively, realizing the layered rendering and display of the interactive object, the background image, and the screen. The image overlay module is to overlay and display the real image with the image of the virtual world to realize the mixed reality effect.

8. The naked eye 3D mixed reality interactive display method according to claim 7, characterized in that: The camera calibration module extracts feature points from a set checkerboard image. According to the camera pinhole model theory, the camera's intrinsic parameter matrix K can be calculated by transforming the camera coordinate system to the pixel coordinate system, which specifically includes: Step A1: Set (u0, v0) as the projection coordinates of the camera optical axis in the pixel coordinate system. Then the transformation of a point p(x, y) in the image coordinate system to the pixel coordinate system can be expressed in homogeneous coordinate form: Among them, dx and dy represent the physical width and height of a pixel respectively; Step A2: A point P(x c ,y c ,z c ) to the image coordinate system point p(x,y) is transformed into the homogeneous coordinates: Where f is the distance between the center of the camera coordinate system and the center of the image coordinate system, that is, the focal length; Step A3: The transformation relationship from the camera coordinate system to the pixel coordinate system can be expressed as: Solving the correspondence between feature points can obtain the camera intrinsic parameter matrix K; The camera pose estimation module calculates the camera pose and uses a direct linear transformation algorithm to obtain the transformation relationship between the world coordinate system and the camera coordinate system, and solves the transformation rotation matrix R and the translation matrix T, specifically including: Step B1: The camera coordinate system matrix obtained by the camera calibration module can be used to obtain the conversion relationship between the world coordinate system and the camera coordinate system: From the above formula, we can know that two equations can be constructed for each feature point (3D-2D coordinate pair). When the number of feature points N≥6, a coordinate transformation equation group Ax=0 can be constructed, where A is a matrix composed of multiple groups of coordinate points. This equation group can be solved. Using the DLT algorithm, there are 12 unknowns in [R|T], x=[a1,…,a 12 ] T , the above formula can be changed into: Step B2: Use the least squares method to solve the equation system Ax=0 in step B1, and obtain the least squares solution under the constraint of |x|=1: argmin||Ax|| 2 , performing singular value decomposition on A yields: SVD(A)=USV T ; Among them, A is an m×n matrix, U is an m×m orthogonal matrix, S is an m×n non-negative diagonal matrix, and V is an n×n orthogonal matrix; The last column of the matrix V in step B3 and step B2 is the solution to the desired equation system Ax=0, because: A = [KR KT]; Then the rotation matrix R and translation matrix T are obtained: During the virtual reality screen fusion calculation process, first obtain the virtual screen image position c and angle f; From the camera pose calculation, we know that the virtual camera position C is: C = c*T; The virtual camera angle F is: F = f*R; The 3D virtual reality scene is displayed on the virtual screen, and the virtual world scene is interacted with through the virtual interactive light pen. The virtual camera fuses the virtual screen image and the virtual world interactive scene to obtain a virtual reality fusion image; During the mixed reality rendering and display process, the mixed reality rendering and display module first renders the real picture captured by the camera on the bottom layer, then renders the background and scenes that do not affect the interaction in the 3D virtual reality interaction system into a virtual screen picture, and renders it on the upper layer of the screen in the real picture, and then renders the interactive 3D objects on the upper layer of the virtual screen picture, and masks the rest, and finally displays a mixed reality rendering picture with the 3D virtual reality interaction scene and the real scene superimposed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 5 to 8 are implemented.

10. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 5 to 8 are implemented.

Images