Multi-display-screen interactive display system and method, storage medium and electronic equipment

Abstract

The invention relates to a multi-display-screen interactive display system and method, a storage medium and electronic equipment, and relates to the technical field of information display, the system comprises VR display equipment and naked eye 3D display equipment, the VR display equipment is used for responding to a first projection screen interactive control event for a first projection screen interactive control, and the naked eye 3D display equipment is used for displaying the first projection screen interactive control event. Displaying a first scene picture in the VR display device to the naked eye 3D display device; or, in response to a first instruction interaction event for the first instruction control component, synchronizing the first operation instruction to the naked eye 3D display device; the naked eye 3D display device is used for displaying a second scene picture in the naked eye 3D display device to the VR display device in response to a second screen projection interaction control event for the second screen projection interaction control; or in response to a second instruction interaction event for the second instruction control component, synchronizing the second operation instruction to the VR display equipment. According to the invention, picture sharing between the VR display device and the naked-eye 3D display device is realized.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of information display technology. Specifically, the present disclosure relates to an interactive display system for multiple display screens, an interactive display method for multiple display screens, a computer-readable storage medium, and an electronic device. Background Art

[0002] In existing information display methods, it is impossible to achieve screen sharing between a VR display device and a naked-eye 3D display device.

[0003] It should be noted that the information disclosed in the above background art is only used to enhance the understanding of the background of the present disclosure. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0004] An object of the present disclosure is to provide an interactive display system for multiple display screens, an interactive display method for multiple display screens, a computer-readable storage medium, and an electronic device, so as to at least overcome to some extent the problem that screen sharing between a VR display device and a naked-eye 3D display device cannot be achieved due to limitations and defects of related technologies.

[0005] According to one aspect of the present disclosure, there is provided an interactive display system for multiple display screens, including:

[0006] A VR display device and a naked-eye 3D display device, where the VR display device is communicatively connected to the naked-eye 3D display device;

[0007] The VR display device includes a first screen mirroring interaction control and / or a first instruction control component, and the naked-eye 3D display device includes a second screen mirroring interaction control and / or a second instruction control component;

[0008] The VR display device is configured to respond to a first screen mirroring interaction control event for the first screen mirroring interaction control, and display a first target scene image in the VR display device on the naked-eye 3D display device; or, respond to a first instruction interaction event for the first instruction control component, and synchronize a first operation instruction to the naked-eye 3D display device;

[0009] The naked-eye 3D display device is configured to respond to a second screen mirroring interaction control event for the second screen mirroring interaction control, and display a second target scene image in the naked-eye 3D display device on the VR display device; or, respond to a second instruction interaction event for the second instruction control component, and synchronize a second operation instruction to the VR display device.

[0010] In an exemplary embodiment of the present disclosure, the first operation instruction includes a device attitude operation instruction corresponding to the VR display device and / or a first user operation instruction input through a first external device;

[0011] The second operation instruction includes a somatosensory operation instruction input through a somatosensory interaction method and / or a second user operation instruction input through a second external device.

[0012] In an exemplary embodiment of the present disclosure, the VR display device includes a VR display screen and a first processor, and the autostereoscopic 3D display device includes an autostereoscopic 3D display screen and a second processor;

[0013] The first processor is configured to render a first original scene picture based on the first operation instruction to obtain a first target scene picture, and synchronize the first target scene picture to the VR display screen;

[0014] The first processor is further configured to push the first target scene picture to the second processor when receiving a first screen mirroring interaction control event for a first screen mirroring interaction control component;

[0015] The second processor is configured to push the first target scene picture to the autostereoscopic 3D display screen and display the first target scene picture through the autostereoscopic 3D display screen.

[0016] In an exemplary embodiment of the present disclosure, the first processor is further configured to send the first operation instruction to the second processor when detecting a first instruction interaction event for a first instruction control component;

[0017] The second processor is configured to render a first original scene picture based on the first operation instruction to obtain a first target scene picture, and synchronize the first target scene picture to the autostereoscopic 3D display screen.

[0018] In an exemplary embodiment of the present disclosure, the first processor includes a first data sending component, and the second processor includes a first data receiving component;

[0019] The first processor is further configured to determine whether the first target scene picture is included in the first data sending component, and when determining that the first target scene picture is included in the data sending component, control the first data sending component to send the first target scene picture to the first data receiving component;

[0020] The first processor is further configured to, when determining that the first target scene picture is not included in the first data sending component, control the first data sending component to obtain the left-eye and right-eye display pictures from the VR display screen, splice the left-eye and right-eye display pictures to obtain the first target scene picture, and then send the first target scene picture to the first data receiving component.

[0021] In an exemplary embodiment of the present disclosure, the first processor is further configured to perform reverse distortion on the first picture rendering result corresponding to the first original scene picture to obtain the first target scene picture;

[0022] When determining that the first target scene picture is not included in the first data sending component, the first processor is further configured to perform forward distortion on the picture splicing result corresponding to the left-eye and right-eye display pictures to obtain the first target scene picture.

[0023] In an exemplary embodiment of the present disclosure, during the process of performing reverse distortion on the first picture rendering result, it is implemented in the following manner:

[0024] Split the first picture rendering result into left-eye and right-eye view to obtain the first left-eye picture rendering result and the first right-eye picture rendering result;

[0025] Determine the first left-eye center point position of the first left-eye picture rendering result and the first right-eye center point position of the first right-eye picture rendering result, and determine the first normal image center point distance according to the first left-eye center point position and the first right-eye center point position;

[0026] Determine the first distorted image center point distance according to the first normal image center point distance, and perform reverse distortion on the picture rendering result according to the first distorted image center point distance to obtain the first target scene picture.

[0027] In an exemplary embodiment of the present disclosure, during the process of performing forward distortion on the picture splicing result, it is implemented in the following manner:

[0028] Determine the second left-eye center point position of the second left-eye display picture in the picture splicing result, and the second right-eye center point position of the second right-eye display picture in the picture splicing result, and determine the second distorted image center point distance according to the second left-eye center point position and the second right-eye center point position;

[0029] Determine the second normal image center point distance according to the second distorted image center point distance, and perform forward distortion on the picture splicing result according to the second normal image center distance to obtain the first target scene picture.

[0030] In an exemplary embodiment of the present disclosure, the first processor further includes a first virtual camera and a first network communication component; the second processor further includes a second virtual camera and a second network communication component;

[0031] The first virtual camera is configured to obtain the current device pose data in the device pose operation instruction, and when detecting a first instruction interaction event for the first instruction control component, send the current device pose data to the second network communication component via the first network communication component;

[0032] The second network communication component is configured to synchronize the received current device pose data to the second processor;

[0033] The second processor is configured to update the current camera pose of the second virtual camera according to the current device pose data, and render the first original scene picture according to the updated camera pose data to obtain a first target scene picture.

[0034] In an exemplary embodiment of the present disclosure, the first virtual camera is further configured to obtain the first current lens pose data in the first user operation instruction, and send the first current lens pose data to the second network communication component via the first network communication component;

[0035] The second network communication component is configured to synchronize the received first current lens pose data to the second processor;

[0036] The second processor is configured to update the current camera pose of the second virtual camera according to the first current lens pose data, and render the first original scene picture according to the updated camera pose to obtain a first target scene picture.

[0037] In an exemplary embodiment of the present disclosure, the first current lens pose data includes the unique object identifier of the target virtual object and the first virtual object view angle of the target virtual object after executing the first user operation instruction.

[0038] In an exemplary embodiment of the present disclosure, the second processor is further configured to: determine a first out-of-screen scene picture and a first in-screen scene picture included in the first target scene picture, and determine a first picture ratio occupied by the first out-of-screen scene picture and a second picture ratio occupied by the first in-screen scene picture;

[0039] Determine the camera type of the second virtual camera according to the first picture ratio and the second picture ratio; wherein, the camera type includes at least one of a converging camera, a parallel optical axis camera, and an off-axis camera.

[0040] In an exemplary embodiment of the present disclosure, if the first screen ratio is less than the second screen ratio, it is determined that the camera type is a converging camera or a parallel optical axis camera; if the first screen ratio is greater than the second screen ratio, it is determined that the camera type is an off-axis camera.

[0041] In an exemplary embodiment of the present disclosure, the second processor is further configured to: obtain first current device attitude data of the VR display device at the current moment and first historical device attitude data corresponding to the previous moment of the current moment, and determine a first distance difference between the first current device attitude data and the first historical device attitude data;

[0042] When it is determined that the first distance difference is less than or equal to a preset distance threshold, perform planar linear interpolation on the first current device attitude data according to the first historical device attitude data to obtain a first device attitude interpolation result, and render the first original scene picture based on the first device attitude interpolation result to obtain a first target scene picture;

[0043] When it is determined that the first distance difference is greater than the preset distance threshold, render the first original scene picture based on the first current device attitude data to obtain a first target scene picture.

[0044] In an exemplary embodiment of the present disclosure, the second processor is further configured to: obtain first current lens attitude data of the target virtual object at the current moment and first historical lens attitude data corresponding to the previous moment of the current moment, and determine whether the attitude of the target virtual object rotates according to the first current lens attitude data and the first historical lens attitude data;

[0045] When it is determined that the attitude of the target virtual object rotates, perform spherical linear interpolation on the first current lens attitude data according to the first historical lens attitude data to obtain a first lens attitude interpolation result, and render the first original scene picture based on the first lens attitude interpolation result to obtain a first target scene picture;

[0046] When it is determined that the attitude of the target virtual object remains unchanged, render the first original scene picture according to the first current lens attitude data to obtain a first target scene picture.

[0047] According to one aspect of the present disclosure, there is provided an interactive display method for a multi-display screen, including:

[0048] In response to a first screen mirroring interaction control event for a first screen mirroring interaction control widget, display a first target scene picture in the VR display device on the autostereoscopic 3D display device and display the first target scene picture; or

[0049] In response to a first instruction interaction event for a first instruction control component, synchronize a first operation instruction to the naked-eye 3D display device, and display a first target scene picture rendered according to the first operation instruction; or

[0050] Receive a second target scene picture sent by the naked-eye 3D display device when detecting a second screen mirroring interaction control event for a second screen mirroring interaction control component, and display the second target scene picture; or

[0051] Receive a second operation instruction sent by the naked-eye 3D display device when detecting a second instruction interaction event for a second instruction control component, and display a second target scene picture rendered according to the second operation instruction.

[0052] According to one aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the interactive display method of a multi-display screen described in any one of the above is implemented.

[0053] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0054] A processor; and

[0055] A memory for storing executable instructions of the processor;

[0056] Wherein, the processor is configured to execute the interactive display method of a multi-display screen described in any one of the above by executing the executable instructions.

[0057] In the interactive display system of a multi-display screen provided by the embodiments of the present disclosure, by the VR display device in response to a first screen mirroring interaction control event for a first screen mirroring interaction control component, display a first scene picture in the VR display device to the naked-eye 3D display device; or, in response to a first instruction interaction event for a first instruction control component, synchronize a first operation instruction to the naked-eye 3D display device; it is also possible to pass the naked-eye 3D display device in response to a second screen mirroring interaction control event for a second screen mirroring interaction control component, display a second scene picture in the naked-eye 3D display device to the VR display device; or, in response to a second instruction interaction event for a second instruction control component, synchronize a second operation instruction to the VR display device, realizing picture sharing between the VR display device and the naked-eye 3D display device.

[0058] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Description of the Drawings

[0059] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0060] Figure 1 Schematic diagram showing a structural example of an interactive display system with multiple display screens according to an exemplary embodiment of the present disclosure.

[0061] Figure 2 Schematic diagram showing a structural example of the implementation principle of an interactive display system with multiple display screens according to an exemplary embodiment of the present disclosure.

[0062] Figure 3 Schematic diagram showing a scenario example of the specific transmission process of a first target scene screen according to an exemplary embodiment of the present disclosure.

[0063] Figure 4 Schematic diagram showing a structural principle example of a first data sending component according to an exemplary embodiment of the present disclosure.

[0064] Figure 5 Schematic diagram showing a structural principle example of a first data receiving component according to an exemplary embodiment of the present disclosure.

[0065] Figure 6 Schematic diagram showing a specific scenario example of the distortion and warping of the screen content according to an exemplary embodiment of the present disclosure.

[0066] Figure 7 Schematic diagram showing a scenario example of the reverse distortion process of the screen content according to an exemplary embodiment of the present disclosure.

[0067] Figure 8 Schematic diagram showing an example of coordinate system conversion according to an exemplary embodiment of the present disclosure.

[0068] Figure 9 Schematic diagram showing a scenario example of the forward distortion process of the screen content according to an exemplary embodiment of the present disclosure.

[0069] Figure 10 Schematic diagram showing an example of the camera type according to an exemplary embodiment of the present disclosure.

[0070] Figure 11 Schematic diagram showing an electronic device for implementing an interactive display method with multiple display screens according to an exemplary embodiment of the present disclosure. Detailed implementation manners

[0071] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will recognize that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or may be implemented using other methods, components, devices, steps, etc. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0072] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0073] With the rapid development of virtual reality (VR) technology and its unique sense of immersion and interaction methods, it has been widely applied in various fields; for example, virtual reality technology can be applied to entertainment, education, industrial design, and remote training, etc., in many different technical fields. At the same time, with the entry of some high-tech enterprises, the development of virtual reality has been further accelerated, making products with virtual reality functions (such as head-mounted virtual reality devices) increasingly mature; at the same time, the light field displays that have emerged in recent years can also bring a sense of immersive experience to people and are gradually accepted by the majority of users. In the actual application process, head-mounted virtual reality devices are only suitable for single-person experience and cannot share the experience content with others; however, if the virtual reality device and the light field display can be combined, the scene content displayed in the virtual reality device can be shared with the light field display, so as to achieve the purpose of multi-user interconnection and interoperability of the same scene content, and can greatly improve the user experience and promote the continuous development of the two types of devices.

[0074] Based on this, the example embodiments of the present disclosure first provide an interactive display system with multiple display screens. Specifically, refer to Figure 1As shown in the figure, the interactive display system of the multi-display screen may include a VR display device 110 and a naked-eye 3D display device 120; among them, the VR display device may be communicatively connected to the naked-eye 3D display device through a wired network or a wireless network; of course, the VR display device may also be communicatively connected to the naked-eye 3D display device through a local area network, or communicatively connected to the naked-eye 3D display device through a serial bus. This example does not make special restrictions on this; in the process of actual application, the VR display device may include a first screen mirroring interaction control and / or a first instruction control component, and the naked-eye 3D display device includes a second screen mirroring interaction control and / or a second instruction control component; at the same time, in the process of interactive display, the VR display device may be used to respond to a first screen mirroring interaction control event for the first screen mirroring interaction control, and display a first target scene picture in the VR display device to the naked-eye 3D display device; or, in response to a first instruction interaction event for the first instruction control component, synchronize a first operation instruction to the naked-eye 3D display device; the naked-eye 3D display device may be used to respond to a second screen mirroring interaction control event for the second screen mirroring interaction control, and display a second target scene picture in the naked-eye 3D display device to the VR display device; or, in response to a second instruction interaction event for the second instruction control component, synchronize a second operation instruction to the VR display device.

[0075] In the interactive display system of the multi-display screen described above, the VR display device may be used to respond to a first screen mirroring interaction control event for the first screen mirroring interaction control, and display a first scene picture in the VR display device to the naked-eye 3D display device; or, in response to a first instruction interaction event for the first instruction control component, synchronize a first operation instruction to the naked-eye 3D display device; the naked-eye 3D display device may also be used to respond to a second screen mirroring interaction control event for the second screen mirroring interaction control, and display a second scene picture in the naked-eye 3D display device to the VR display device; or, in response to a second instruction interaction event for the second instruction control component, synchronize a second operation instruction to the VR display device, realizing the screen sharing between the VR display device and the naked-eye 3D display device.

[0076] Hereinafter, the interactive display system of the multi-display screen described in the exemplary embodiments of the present disclosure will be further explained and described with reference to the accompanying drawings.

[0077] First, the terms involved in the exemplary embodiments of the present disclosure will be explained and described.

[0078] Virtual Reality (VR): A three-dimensional environment generated by a computer that allows users to experience and interact with the virtual world through sensory input devices such as vision and hearing (such as VR headsets and controllers). VR emphasizes immersion, making users feel as if they are in the virtual environment, and is commonly used in fields such as gaming, education, and training.

[0079] 3D Real-Time Rendering: It refers to the process of quickly generating three-dimensional images by a computer while the user is operating; in the actual application process, 3D real-time rendering can be used to dynamically calculate effects such as lighting, shadows, and materials so that users can immediately see the interaction effects, and it can be widely applied to scenarios such as games, virtual reality, and simulation.

[0080] Networking Communication: It refers to the process of data exchange between two or more devices through wired or wireless networks; in the actual application process, networking communication can adopt various protocols and technologies, such as TCP / IP protocol, HTTP protocol, and WebSocket protocol, etc. This networking communication can support from simple message passing to complex data synchronization, including various applications on the Internet, local area network, and wireless networks.

[0081] Video Streaming: It refers to the technology of real-time transmitting video content, which can send video data to multiple display terminals in real time through the way of network streaming; in the actual application process, video streaming technology includes processes such as video encoding (compressing video data), video transmission (which can be achieved through protocols such as RTMP and HLS), and video decoding (the client decompresses video data), and is commonly used in different scenarios such as live broadcast, video conferencing, and online content distribution.

[0082] 3D Game Engine: It is a software platform that provides tools and frameworks required for developing three-dimensional games, including multiple modules such as graphics rendering, physical simulation, sound processing, and artificial intelligence; in the actual application process, 3D game engines simplify the game development process. Common 3D game engines can include but are not limited to Unity and Unreal Engine, etc., which support multi-platform publishing and provide rich plugins and resources.

[0083] VR Headset: It is a wearable device designed specifically for experiencing virtual reality content, usually equipped with a display screen, sensors, and audio output devices; in the actual application process, VR headsets can cover the user's field of vision and track the movements of the user's head and hands, thereby achieving the purpose of providing users with an immersive visual and auditory experience.

[0084] Secondly, the technical implementation principle of the exemplary embodiments of the present disclosure will be explained and described. Specifically, the interactive display system of multiple display screens recorded in the exemplary embodiments of the present disclosure realizes real-time linkage between a VR display device and a naked-eye 3D display device; in the actual application process, the scene content (i.e., the target scene picture) displayed in the VR display device can be synchronously transferred to the naked-eye 3D display device in real time, so that viewers can experience 3D content without wearing VR devices, thereby achieving the purpose of sharing the screen between the VR display device and the naked-eye 3D display device; moreover, based on the interactive display system of multiple display screens recorded in the exemplary embodiments of the present disclosure, the naked-eye 3D display device can be used as a powerful supplement to the performance level of the VR display device, and at the same time, the display content of the naked-eye 3D display device is enriched, solving problems such as insufficient display resources.

[0085] In one exemplary embodiment, the interactive display system of multiple display screens recorded in the exemplary embodiments of the present disclosure can be implemented in two ways during the actual interactive display process, namely, by sharing the screen or sharing operation instructions. Among them, in the process of displaying the first target scene picture in the VR display device on the naked-eye 3D display device, it can be achieved in the following ways: One implementation method is that when the VR display device detects a first screen mirroring interaction control event for the first screen mirroring interaction control component, it can display the first target scene picture in the VR display device on the naked-eye 3D display device, thus realizing screen sharing; another implementation method is that when the VR display device detects a first instruction interaction event for the first instruction control component, it can synchronize the first operation instruction to the naked-eye 3D display device, and then perform screen rendering on the naked-eye 3D display device based on the first operation instruction, and display the rendered first target scene picture; that is to say, the screen of the VR display device can be mirrored to the naked-eye 3D display device by one-key screen mirroring, or the operation instruction of the VR display device can be mirrored to the naked-eye 3D display device by one-key sending an instruction, and the rendering and display are performed on the naked-eye 3D display device.

[0086] Further, in the process of displaying the second target scene screen in the naked-eye 3D display device to the VR display device, it can be achieved in the following ways: One implementation method is that when the naked-eye 3D display device detects a second screen mirroring interaction control event for the second screen mirroring interaction control component, it can display the second target scene screen in the naked-eye 3D display device to the VR display device side; Another implementation method is that when the naked-eye 3D display device detects a second instruction interaction event for the second instruction control component, it can synchronize the second operation instruction to the VR display device side, and then perform screen rendering on the VR display device side based on the second operation instruction, and display the rendered second target scene screen; That is to say, the screen of the naked-eye 3D display device side can be screen-mirrored to the VR display device side by one-key screen mirroring, or the operation instruction of the naked-eye 3D display device side can be screen-mirrored to the VR display device side by one-key sending an instruction, and rendering and display are performed on the VR display device side.

[0087] That is to say, in the process of screen sharing between the VR display device side and the naked-eye 3D display device side, the screen of the VR display device side can be screen-mirrored to the naked-eye 3D display device side, or the screen of the naked-eye 3D display device side can be screen-mirrored to the VR display device side; Or, the operation instruction of the VR display device side can also be displayed to the naked-eye 3D display device side, and the operation instruction of the naked-eye 3D display device side can also be displayed to the naked-eye 3D display device side (the naked-eye 3D display device side can also be controlled by the VR display device side, or the VR display device side can be controlled by the naked-eye 3D display device side). In the actual application process, if screen mirroring is required, it can be directly achieved through the screen mirroring control component; If operation instruction sharing is required, when the VR display device side needs to be the master control party, it can send a control instruction to the naked-eye 3D display device side based on the first screen mirroring interaction control component. When the naked-eye 3D display device side receives the control instruction, it can stop its own ongoing instruction input operation and receive the first operation instruction sent by the VR display device side; Further, when the naked-eye 3D display device side needs to be the master control party, it can send a control instruction to the VR display device side based on the second screen mirroring interaction control component. When the VR display device side receives the control instruction, it can stop its own ongoing instruction input operation and receive the second operation instruction sent by the naked-eye 3D display device side. Based on this, not only is screen sharing achieved, but also the user experience of multi-terminal users is improved; That is, the users of the VR display device side and the users of the naked-eye 3D display device side can not only watch, but also experience.

[0088] It should be noted here that in the actual application process, a screen mirroring application client or an instruction control application client can be installed on the VR display device side and the naked-eye 3D display device side respectively, or the corresponding function modules can be installed. Or, the corresponding function programs can be set on the VR display device side and the naked-eye 3D display device side in a pluggable manner. This example does not make special restrictions on this. Further, in some other possible implementation manners, the corresponding buttons can also be added or reduced in the supporting external device (such as a handle) to achieve this.

[0089] Hereinafter, with reference to specific embodiments, Figure 1 the interactive display system of the multi-display screen shown in

[0090] Figure 2 A structural example diagram schematically showing the implementation principle of an interactive display system of a multi-display screen according to an exemplary embodiment of the present disclosure is shown. Specifically, referring to Figure 2 as shown, the above-mentioned VR display device may further include a VR display screen 201 and a first processor 202, and the above-mentioned naked-eye 3D display device may further include a naked-eye 3D display screen 203 and a second processor 204; at the same time, the first processor is communicatively connected to the VR display screen and the second processor respectively, and the second processor is communicatively connected to the naked-eye 3D display screen; that is, in the actual application process, the VR display device may include a VR display screen rendered by a high-end graphics card based on a computer and a first processor with an independent rendering function, and the first processor can be implemented by means of an embedded SOC chip; and, by placing an SDK in the SOC chip, while the rendering result is displayed on the head-mounted display (that is, the VR display screen), the picture can be video-encoded and sent to the second processor through network communication. It should also be noted here that in the process of sharing the screen of the VR display device and the naked-eye 3D display device, not only can the screen be directly shared, but also the instructions can be directly shared. Each of these two methods has its own advantages and disadvantages, and in the actual application process, the corresponding implementation method can be selected according to actual needs. This example does not make special restrictions on this.

[0091] In an exemplary embodiment, the advantages of the screen sharing described above are as follows: there is no need to consider the rendering capabilities of the devices. However, this method also has the following disadvantages: on the one hand, the resolutions are inconsistent. That is, the monocular resolution of the VR display screen is generally square, for example, it can be 2K*2K or 4K*4K, etc. However, the resolution of the autostereoscopic display screen is generally rectangular, so there is a problem of mismatched display ratios, which reduces the user experience of the users viewing the autostereoscopic display screen. On the other hand, the first field of view angle of the first virtual camera in the VR display device and the second field of view angle of the second virtual camera in the autostereoscopic display device are inconsistent. Specifically, in the actual application process, the angle of the first field of view angle of the VR display device is often relatively large, which is between 90 degrees and 120 degrees. However, the optimal viewing angle of the second field of view angle of the autostereoscopic display device is much smaller, which is within 30 to 45 degrees. Therefore, if the target scene screen in the VR display device is directly displayed on the autostereoscopic display device, there will be a problem of mismatched field of view angles, which further reduces the user experience of the users viewing the autostereoscopic display screen. On the other hand, the rotation speed of the VR headset is very fast, which will cause the problem of discontinuous and jittery images on the autostereoscopic display device. And since the rendering result is an image, it is very difficult to achieve smooth interpolation and transition, which will affect the viewing experience.

[0092] In an exemplary embodiment, although the implementation method of the instruction sharing described above can solve the disadvantages existing in the first implementation method, it itself also has the following disadvantages: on the one hand, corresponding application programs need to be set in the VR display device and the autostereoscopic display device respectively, which imposes a relatively large burden on the memory systems of the autostereoscopic display device and the VR display device. On the other hand, it has relatively high requirements for the real-time rendering capabilities of the autostereoscopic display device and the VR display device, and increases the computing burden on the autostereoscopic display device and the VR display device.

[0093] In an exemplary embodiment, if the first target scene screen is shared by means of screen sharing, it can be achieved in the following manner: The first processor renders the first original scene screen based on the first operation instruction to obtain the first target scene screen, and synchronizes the first target scene screen to the VR display screen; the first processor can also push the first target scene screen to the second processor when receiving the first screen projection interaction control event for the first screen projection interaction control widget; the second processor pushes the first target scene screen to the autostereoscopic display screen and displays the first target scene screen through the autostereoscopic display screen. Wherein, the first operation instruction recorded here may include a device posture operation instruction corresponding to the VR display device and / or a first user operation instruction input through the first external device; further, the device posture operation instruction refers to an instruction for the user to move the VR device left or right, up or down, or rotate; the first user operation instruction refers to an operation instruction input by the user through the first external device (such as a handle matching the VR display device), such as controlling the corresponding virtual object to move left or right, up or down, or rotate, etc., and may also include controlling the target virtual object to release a certain skill, etc. This example does not make special restrictions on this; of course, the user operation instruction recorded here can not only act on the virtual object in the game scene, but also act on other objects in other scenes, such as various teaching aids in the teaching scene, etc. This example does not make special restrictions on this. In the actual application process, if the first processor receives the corresponding operation instruction, it can render a new scene screen based on the operation instruction and display it. It should be further supplemented here that in the process of pushing the first target scene screen to the VR display screen, it can be pushed by means of pushing the left and right eye videos separately.

[0094] In an example embodiment, if the second target scene screen is shared by screen sharing, it can be achieved in the following way: If the second target scene screen is shared by screen sharing, it can be achieved in the following way: the second processor renders the second original scene screen based on the second operation instruction to obtain the second target scene screen, and synchronizes the second target scene screen to the naked-eye 3D display screen; the second processor can also push the second target scene screen to the first processor when receiving the second projection screen interaction control event for the second projection screen interaction control; the first processor pushes the second target scene screen to the VR display screen, and displays the first target scene screen through the VR display screen. Among them, the second operation instruction recorded here may include a somatosensory operation instruction input through somatosensory interaction, and a second user operation instruction input through a second external device, etc. Further, the somatosensory operation instruction refers to an instruction for the user to interact with the naked eye 3D device left and right, up and down, or rotate through somatosensory means; the second user operation instruction refers to an operation instruction input by the user through a second external device (such as an eye tracking system, a gesture recognition system, a mouse, and a keyboard, etc. that are matched with the naked eye 3D display device); for example, in a game scene, the corresponding virtual object can be controlled to move left and right, up and down, or rotate, etc., and can also include controlling the target virtual object to release a certain skill, etc., and this example does not impose special restrictions on this; of course, the user operation instruction recorded here can act not only on virtual objects in the game scene, but also on other objects in other scenes, such as various teaching aids in teaching scenes, etc., and this example does not impose special restrictions on this. In the actual application process, if the second processor receives the corresponding operation instruction, a new scene screen can be rendered based on the operation instruction and displayed.

[0095] It should be noted that the first original scene picture and the second original scene picture, the first target scene picture and the second target scene picture recorded here may be the same or different. This example is only for the convenience of explanation and there is no special limitation on this.

[0096] Figure 3 The following schematically shows a scene example diagram of a specific transmission process of a target scene picture according to an exemplary embodiment of the present disclosure. Figure 3 As shown, the first processor may further include a first data sending component 301, and the second processor may include a first data receiving component 302; wherein the specific structural example diagrams of the first data sending component and the first data receiving component may refer to Figure 4 as well as Figure 5 Further, by Figure 4From the structural example diagram of the first data sending component shown, it can be known that the first data sending component can be used for content generation and interaction control. At the same time, the first data sending component can include device broadcasting (for broadcasting its own port number to facilitate establishing a connection with the data receiving component), device discovery (discovering connectable devices), connection establishment (establishing a connection between the sending end and the receiving end), connection authentication (authenticating the device security of the receiving end), capability negotiation (such as the supported audio and video frame rates and resolutions), content acquisition, video encoding, audio encoding, content transmission (transmitting the corresponding audio and video data, which can be implemented based on RTSP), keep-alive mechanism (determining that the receiving end is online), TCP / UDP switching, buffer queue (data can be put into the buffer queue waiting for transmission), reverse control, frame skipping processing, and connection closing, etc. In the actual application process, the data sending component can be compatible with various different screen mirroring protocols such as Miracast, AirPlay, WiDi, GoogleCast, DLNA, etc. At the same time, it can also adapt to different operating systems of Android, Windows, iOS, MacOS, and Linux, support resolution and frame rate negotiation, and facilitate the invocation and integration of third-party partners.

[0097] Furthermore, from Figure 5 From the structural example diagram of the first data receiving component shown, it can be known that the first data receiving component can be used for content display and interaction control. At the same time, the first data receiving component can include device broadcasting (for broadcasting its own port number to facilitate establishing a connection with the data sending component), device discovery (discovering connectable devices), connection establishment (establishing a connection between the receiving end and the sending end), connection authentication (authenticating the device security of the sending end), capability negotiation (such as the supported audio and video frame rates and resolutions), content rendering (which can be implemented based on the SideBySide method), video decoding, audio decoding, content reception (receiving the corresponding audio and video data, which can be implemented based on RTSP), keep-alive mechanism (determining that the sending end is online), TCP / UDP switching, buffer queue (data can be put into the buffer queue waiting for reception), reverse control, frame skipping processing, and connection closing, etc. In the actual application process, the data receiving component can be compatible with various different screen mirroring protocols such as Miracast, Airplay, WiDi, GoogleCast, DLNA (Digital Living Network Alliance), etc. At the same time, it can also perform wired screen mirroring through wireless WiFi or USB data cable (the USB debugging option needs to be turned on for Android devices), the SDK provides audio, video, and interaction data, and the display interface needs to be developed by the integrating party. Among them, the specific connection methods that the data sending component and the data receiving component can support are shown in Table 1 below:

[0098] Table 1

[0099]

[0100] It should also be noted here that when the second target scene screen is displayed on the VR display device, the first data receiving component described above can also be used as a data sending component, and the first data sending component can also be used as a data receiving component. No special restrictions are imposed on this example.

[0101] In an exemplary embodiment, when sending and receiving the first target scene screen based on the first data sending component and the first data receiving component, it can be implemented in the following manner: The first processor determines whether the first target scene screen is included in the first data sending component, and when it is determined that the first target scene screen is included in the first data sending component, controls the first data sending component to send the first target scene screen to the first data receiving component; The first processor is also used to control the first data sending component to obtain the left and right eye display screens from the VR display screen, splice the left and right eye display screens to obtain the first target scene screen, and then send the first target scene screen to the first data receiving component when it is determined that the first target scene screen is not included in the first data sending component. That is, in the actual application process, if the corresponding scene screen is not included in the first data sending component, it is necessary to obtain the first target scene screen currently displayed from the VR display screen, and then splice and feedback it to the first data receiving component.

[0102] In an exemplary embodiment, the first processor is also used to perform reverse distortion on the first frame rendering result corresponding to the first original scene screen to obtain the first target scene screen. That is, in the actual application process, when the VR display device displays 3D content, due to the physical structure lens, the target scene screen displayed will be distorted; among them, for the specific scene example diagram of the distortion, reference can be made to Figure 6 as shown. Therefore, in order to avoid distortion, it is necessary to add reverse distortion processing to the first target scene screen. When the processed screen after reverse distortion is mapped to the VR display screen, a forward distortion can be performed again to obtain the first target scene screen that can be normally displayed. Among them, for the scene diagram of the specific reverse distortion process, reference can be made to Figure 7 as shown.

[0103] In an exemplary embodiment, during the process of performing forward distortion on the rendering result of the first frame, it is achieved in the following manner: splitting the rendering result of the first frame into left-eye and right-eye views to obtain the rendering result of the first left-eye frame and the rendering result of the first right-eye frame; determining the position of the center point of the first left-eye frame in the rendering result of the first left-eye frame and the position of the center point of the first right-eye frame in the rendering result of the first right-eye frame, and determining the distance between the center points of the first normal images based on the position of the center point of the first left-eye frame and the position of the center point of the first right-eye frame; determining the distance between the center points of the first distorted images based on the distance between the center points of the first normal images, and performing reverse distortion on the rendering result of the frame based on the distance between the center points of the first distorted images to obtain the first target scene frame. Specifically, in the actual application process, the distortion generated by the physical structure of the VR lens can be implemented based on the radial distortion formula; meanwhile, during the process of performing reverse distortion based on the radial distortion formula, it can be achieved through the following formulas (1) - (6):

[0104] r d = r u (1 - α|r u | 2 ); Formula (1)

[0105]

[0106] x n1 = (2x 1 - w) / w; Formula (3)

[0107] x n2 = (2x 2 - w) / w; Formula (4)

[0108] y n1 = (2y 1 - h) / h; Formula (5)

[0109] y n2 = (2y 2 - h) / h; Formula (6)

[0110] Wherein, r d is the distance between the center points of the first distorted images, r u is the distance between the center points of the first normal images, (x n1 , y n1 ) is the position of the center point of the first left-eye frame after coordinate transformation, (x n2 , y n2 ) is the position of the center point of the first right-eye frame after coordinate transformation, (x 1 , y 1 ) is the position of the center point of the first left-eye frame before coordinate transformation (i.e., the center point position of the rendering result of the first left-eye frame), (x2 , y 2 ), which is the position of the center point of the first right eye before coordinate transformation (i.e., the center point of the rendering result of the first right eye picture). w is the picture width of the first picture rendering result, h is the picture height of the first picture rendering result, and α is the constant coefficient of the lens of the VR display device. At the same time, the reason for coordinate transformation is that r is calculated around the center point of the image where the optical axis is located. If radial distortion is to be performed, coordinate normalization needs to be carried out first to map the coordinate system in the image to the coordinate system in the radial distortion formula, so as to improve the accuracy of the distance of the center point of the first distorted image. Among them, for the scene example diagram of the specific coordinate transformation process, reference can be made to Figure 8 as shown

[0111] In an exemplary embodiment, when it is determined that the first data sending component does not include the first target scene picture, the first processor is further configured to perform forward distortion on the picture splicing result corresponding to the left and right eye display pictures to obtain the first target scene picture. That is, in the actual application process, if the original undistorted target scene picture cannot be obtained, the target scene picture needs to be determined according to the picture splicing result corresponding to the left and right eye display pictures that have been distorted. At the same time, in order to display the undistorted picture on the naked-eye 3D display, forward distortion needs to be performed on the picture splicing result to obtain the target scene picture, so that a picture with a normal ratio can be displayed on the naked-eye 3D display. Among them, for the scene example diagram of the specific forward distortion process, reference can be made to Figure 9 as shown

[0112] In an exemplary embodiment, in the process of performing forward distortion on the picture splicing result, it is implemented in the following manner: determine the position of the center point of the second left eye of the second left eye display picture in the picture splicing result, and the position of the center point of the second right eye of the second right eye display picture in the picture splicing result, and determine the distance of the center point of the second distorted image according to the position of the center point of the second left eye and the position of the center point of the second right eye; determine the distance of the center point of the second normal image according to the distance of the center point of the second distorted image, and perform forward distortion on the picture splicing result according to the distance of the center point of the second normal image to obtain the first target scene picture. Specifically, in the process of performing forward distortion based on the radial distortion formula, it can be implemented through the following formulas (7)-(12):

[0113]

[0114] where r' u is the distance of the center point of the second normal image, r' d is the distance of the center point of the second distorted image, α is the constant coefficient of the lens of the VR display device, (x' n1, y' n1 ) is the position of the center point of the second left eye of the distorted image (i.e., the center point position of the second left-eye display screen), (x' n2 , y' n2 ) is the position of the center point of the second right eye of the distorted image (i.e., the center point position of the second right-eye display screen), (x' 1 , y' 1 ) is the position of the center point of the second left eye of the normal image, (x' 2 , y' 2 ) is the position of the center point of the second right eye of the normal image; w' is the width of the image splicing result, and h' is the height of the image splicing result.

[0115] It should be further noted here that in the process of synchronizing the second target scene image in the naked-eye 3D display device to the VR display device, the specific implementation method is similar to the synchronization process of the first target scene image described above, and no further elaboration will be made here.

[0116] In an exemplary embodiment, if the first target scene image is shared by means of instruction sharing, it can be achieved in the following way: the first processor is used to render the first original scene image based on the operation instruction to obtain the first target scene image, and synchronize the first target scene image to the VR display screen; the first processor is also used to send the first operation instruction to the second processor when detecting a first instruction interaction event for the first instruction control component; the second processor is used to render the first original scene image based on the first operation instruction to obtain the first target scene image, and synchronize the first target scene image to the naked-eye 3D display screen; wherein, the first operation instruction recorded here may include a device attitude operation instruction corresponding to the VR display device and / or a first user operation instruction input through the first external device corresponding to the target virtual object. And, if the first target scene image is shared by means of instruction sharing, it is also necessary to configure a 3D development engine in the second processor, establish a model scene in the 3D development engine, and set scene information such as materials and lighting, so as to ensure that the VR display device side and the naked-eye 3D display device side can adopt exactly the same 3D scene; at the same time, since the VR display device side and the naked-eye 3D display device side need to perform independent image rendering, it is also necessary to synchronize the operation content data in the form of instructions through network communication (i.e., synchronize the first operation instruction to the naked-eye 3D display device side).

[0117] In an exemplary embodiment, the first processor further includes a first virtual camera and a first network communication component; the second processor further includes a second virtual camera and a second network communication component; the first virtual camera is configured to obtain current device pose data in a device pose operation instruction, and when detecting a first instruction interaction event for the first instruction control component, send the current device pose data to the second network communication component via the first network communication component; the second network communication component is configured to synchronize the received current device pose data to the second processor; the second processor is configured to update the current camera pose of the second virtual camera according to the current device pose data, and render a first original scene picture according to the updated camera pose data to obtain a first target scene picture. That is to say, in the actual application process, it is also necessary to import a first virtual camera and a first network communication component into the first processor, and import a second virtual camera and a second network communication component into the second processor; wherein, the first virtual camera may be a VR dedicated virtual camera, and the second virtual camera may be a binocular virtual camera for real-time rendering of binocular stereo pictures; at the same time, the first network communication component and the second network communication component are used to synchronize operation instructions. In the actual application process, the first network communication component may act as a client, and the second network communication component may act as a server to establish a connection; at the same time, during the connection establishment process, the server side may establish a server and wait for a connection, and the client side may be responsible for actively connecting to the server side to establish a connection; after the connection is successful, the client side sends the user's status information (i.e., the first operation instruction), including but not limited to head pose information, rotation coordinates, body position coordinates, and handle control coordinates, etc. as data packets to the server side at a fixed frequency, so as to achieve the purpose of realizing the interaction between virtual reality and naked-eye 3D.

[0118] It should also be supplemented and explained here that during the process of synchronizing the second operation instruction to the VR display device, the first network communication component described above may also act as a server side, and the second network communication component may also act as a client side, and this example does not make special restrictions on this.

[0119] In an exemplary embodiment, the first virtual camera is further configured to obtain the first current camera pose data in the first user operation instruction, and send the first current camera pose data to the second network communication component via the first network communication component; the second network communication component is configured to synchronize the received first current camera pose data to the second processor; the second processor is configured to update the current camera pose of the second virtual camera according to the first current camera pose data, and render the first original scene picture according to the updated camera pose to obtain the first target scene picture; wherein, the first current camera pose data recorded herein includes the unique object identifier of the target virtual object and the first virtual object view of the target virtual object after executing the first user operation instruction. The application program on the PC controls the pose of the virtual camera according to this data, realizes the state synchronization between the first virtual camera and the second virtual camera, so as to restore the display of VR content. That is to say, in the actual application process, for an interactive target virtual object, a unique number ID (i.e., the unique object identifier) can be defined. When the position of the target virtual object changes, rotates, or shrinks due to the first operation instruction, the ID number and the corresponding pose data are sent to the second processor, so that the second processor can modify the position of the target virtual object in the scene picture according to this data, thereby realizing the state synchronization of the target virtual object. Among them, the format definition of the operation instruction can be shown in Table 2 below:

[0120] Table 2

[0121] Instruction Name Data Type Length Operation Object Function SyncPosition Vector3 Floatx3 ObjectId Synchronization Position Coordinates SyncRotation Quaternion Floatx4 ObjectId Synchronization Rotation Attitude SyncScale Vector3 Floatx3 ObjectId Synchronization Scaling Dimension SyncColor Vector3 Floatx3 ObjectMaterialId Synchronization Model Color SyncCameraStat Float Array Float[n] Camera Synchronization Camera Status

[0122] It should be further noted here that in the process of synchronizing the second operation instruction to the VR display device, the specific synchronization process is similar to that of the first operation instruction, and will not be further elaborated here.

[0123] In an exemplary embodiment of the present disclosure, the second processor is further configured to: determine a first out-of-screen scene image and a first in-screen scene image included in the first target scene image, and determine a first image ratio occupied by the first out-of-screen scene image and a second image ratio occupied by the first in-screen scene image; determine the camera type of the second virtual camera according to the first image ratio and the second image ratio; wherein the camera type includes a converging camera, a parallel optical axis camera, an off-axis camera, etc.; and, in the process of determining the camera type, if the first image ratio is less than the second image ratio, determine the camera type as a converging camera or a parallel optical axis camera, and if the first image ratio is greater than the second image ratio, determine the camera type as an off-axis camera. That is, in the actual application process, since the viewing content of the VR display device is often about 1.2 meters in front of the viewer, when presented on the naked-eye 3D screen, a large in-screen effect will be presented; therefore, according to the actual display effect, a converging camera, a parallel optical axis camera, and an off-axis camera can be selected as the second virtual camera; further, for a scene with more in-screen performance, it is recommended to use a parallel optical axis camera or a converging camera; at the same time, for a scene with more out-of-screen content, it is recommended to use an off-axis camera type; and, the converging camera, the parallel optical axis camera, and the off-axis camera can respectively refer to Figure 10 shown in 1001, 1002, and 1003 in

[0124] In an exemplary embodiment of the present disclosure, the second processor is further configured to: obtain first current device pose data of the VR display device at the current moment and first historical device pose data corresponding to the previous moment of the current moment, and determine a first distance difference between the first current device pose data and the first historical device pose data; when it is determined that the first distance difference is less than or equal to a preset distance threshold, perform planar linear interpolation on the first current device pose data according to the first historical device pose data to obtain a first device pose interpolation result, and render the first original scene picture based on the first device pose interpolation result to obtain a first target scene picture; when it is determined that the first distance difference is greater than the preset distance threshold, render the first original scene picture based on the first current device pose data to obtain a first target scene picture. That is, in actual tests, due to the delay and instability of network transmission, there may be jitter and discontinuity of the picture; at the same time, since the moving range and rotation angle of the VR headset are large and there are frequent angle and position adjustments, when transmitted to the autostereoscopic display device, severe jitter and skipping will be felt, resulting in discomfort when viewing on the autostereoscopic screen; therefore, after the second processor receives the camera pose information, smoothing and linear interpolation processing are required. Among them, the specific calculation process of linear interpolation can be shown as the following formula (13):

[0125] P(t) = P 0 + t * (P 1 - P 0 ); Formula (13)

[0126] Wherein, P(t) is the device pose interpolation result, P 0 is the first historical device pose data, P 1 is the first current device pose data, and t is a parameter between 0 and 1, representing the progress of interpolation; for example, during the process of t gradually increasing from 0 to 1, the coordinate point smoothly moves from P 0 to P 1 . At the same time, an additional judgment condition is: because there is a problem of long-distance transmission on the VR side, the position coordinate gap between the front and back frames is very large at this time. If smooth interpolation is still used, a large displacement of the camera picture will be seen, and there will be an intersection with the model; therefore, interpolation is enabled only when it is judged that the distance between the coordinates of this frame and the previous frame is less than the threshold value (that is, the preset distance threshold X), otherwise the coordinates are directly switched.

[0127] In an exemplary embodiment of the present disclosure, the second processor is further configured to: obtain first current camera pose data of the target virtual object at the current moment and first historical camera pose data of the previous moment corresponding to the current moment, and determine whether the pose of the target virtual object has rotated according to the first current camera pose data and the first historical camera pose data; when it is determined that the pose of the target virtual object has rotated, perform spherical linear interpolation on the first current camera pose data according to the first historical camera pose data to obtain a first camera pose interpolation result, and render the first original scene image based on the first camera pose interpolation result to obtain a first target scene image; when it is determined that the pose of the target virtual object remains unchanged, render the first original scene image according to the first current camera pose data to obtain a first target scene image. That is, in a three-dimensional rendering engine, quaternions are usually used to represent the rotation pose of an object; specifically, in the actual application process, when the angles of the front and rear frames change, the spherical linear interpolation method (i.e., SLerp) is used for interpolation. The specific calculation formula of Slerp can be shown as the following formula (14):

[0128]

[0129] where q 1 is the first historical camera pose data, q 2 is the first current camera pose data, the interpolation parameter is t (0 ≤ t ≤ 1), and q is the camera pose interpolation result; at the same time, w = cos -1 (q 1 ·q 2 ), and the "·" here represents the dot product operation; at the same time, in the process of performing spherical linear interpolation based on Slerp, as t changes from 0 to 1, it can smoothly transition from the historical camera pose data to the current camera pose data.

[0130] So far, the interactive display system of multiple display screens described in the exemplary embodiments of the present disclosure has been fully implemented. Based on the foregoing content, it can be known that the interactive display system of multiple display screens described in the exemplary embodiments of the present disclosure not only realizes the screen sharing between the VR display device and the autostereoscopic 3D display device, but also ensures the continuity of displacement and the correctness of the transmission effect on the basis of ensuring the continuity of the display effect and reducing the sense of dizziness.

[0131] In this exemplary embodiment, an interactive display method for a multi-display screen is also provided. This method can run on a VR display device. Of course, those skilled in the art can also run the method of the present disclosure on other platforms according to requirements, and no special limitation is made in this exemplary embodiment. Specifically, the interactive display method for the multi-display screen may include the following steps: in response to a first screen mirroring interaction control event for a first screen mirroring interaction control component, display a first target scene image in the VR display device on the autostereoscopic 3D display device and display the first target scene image; or, in response to a first instruction interaction event for a first instruction control component, synchronize a first operation instruction to the autostereoscopic 3D display device and display a first target scene image rendered according to the first operation instruction; or, receive a second target scene image sent by the autostereoscopic 3D display device when detecting a second screen mirroring interaction control event for a second screen mirroring interaction control component, and display the second target scene image; or, receive a second operation instruction sent by the autostereoscopic 3D display device when detecting a second instruction interaction event for a second instruction control component, and display a second target scene image rendered according to the second operation instruction.

[0132] In the interactive display method for the multi-display screen described above, it is possible to display a first scene image on the autostereoscopic 3D display device by responding to a first screen mirroring interaction control event for a first screen mirroring interaction control component; or, synchronize a first operation instruction to the autostereoscopic 3D display device by responding to a first instruction interaction event for a first instruction control component; it is also possible to receive a second target scene image sent by the autostereoscopic 3D display device when detecting a second screen mirroring interaction control event for a second screen mirroring interaction control component and display the second target scene image; or, receive a second operation instruction sent by the autostereoscopic 3D display device when detecting a second instruction interaction event for a second instruction control component and display a second target scene image rendered according to the second operation instruction, thereby realizing image sharing between the VR display device and the autostereoscopic 3D display device.

[0133] It should be noted that although several modules or units of a device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0134] In addition, although the steps of the methods in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.

[0135] In an exemplary embodiment of the present disclosure, there is also provided an electronic device capable of implementing the above method. Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method, or a program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as a circuit, a module, or a system here.

[0136] The following refers to Figure 11 to describe the electronic device 1100 according to this embodiment of the present disclosure. Figure 11 The illustrated electronic device 1100 is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0137] As Figure 11 shown, the electronic device 1100 is presented in the form of a general computing device. The components of the electronic device 1100 may include, but are not limited to: at least one of the above-mentioned processing units 1110, at least one of the above-mentioned storage units 1120, a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110), and a display unit 1140.

[0138] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification. For example, the processing unit 1110 can execute the above-mentioned interactive display method for multiple display screens.

[0139] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 11201 and / or a cache storage unit 11202, and may further include a read-only storage unit (ROM) 11203.

[0140] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205. Such program modules 11205 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0141] The bus 1130 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0142] The electronic device 1100 may also communicate with one or more external devices 1200 (such as a keyboard, a pointing device, a Bluetooth device, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be carried out through the input / output (I / O) interface 1150. Moreover, the electronic device 1100 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 1160. As shown in the figure, the network adapter 1160 communicates with other modules of the electronic device 1100 through the bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0143] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or can be implemented by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (such as a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0144] In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of this specification. In some possible implementation manners, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments described in the "Exemplary Method" section shown in this specification.

[0145] The program product for implementing the above method according to an embodiment of the present disclosure may be a portable compact disc read-only memory (CD-ROM) and includes program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0146] The program product may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0147] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal may take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0148] The program code contained on the readable medium may be transmitted by any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0149] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0150] In addition, the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes may be executed synchronously or asynchronously in, for example, multiple modules.

[0151] Other embodiments of the present disclosure will be readily apparent to those skilled in the art after considering the specification and practicing the invention herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not invented by the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are pointed out by the claims.

Claims

1. An interactive display system for multiple display screens, characterized in that: include: A VR display device and a naked-eye 3D display device, wherein the VR display device is communicatively connected with the naked-eye 3D display device; The VR display device includes a first screen projection interaction control and / or a first command control component, and the naked-eye 3D display device includes a second screen projection interaction control and / or a second command control component; The VR display device is used to display the first target scene image in the VR display device to the naked-eye 3D display device in response to a first screen projection interaction control event for a first screen projection interaction control; Alternatively, in response to a first instruction interaction event for a first instruction control component, synchronizing the first operation instruction to the naked-eye 3D display device; The naked-eye 3D display device is used to display the second target scene image in the naked-eye 3D display device to the VR display device in response to a second projection interaction control event for a second projection interaction control control; or, in response to a second command interaction event for a second command control component, synchronize the second operation instruction to the VR display device.

2. The interactive display system of multiple display screens according to claim 1, characterized in that: The first operation instruction includes a device posture operation instruction corresponding to the VR display device, and / or a first user operation instruction input through a first external device; The second operation instruction includes a somatosensory operation instruction input through a somatosensory interaction mode and / or a second user operation instruction input through a second external device.

3. The interactive display system of multiple display screens according to claim 2, characterized in that: The VR display device includes a VR display screen and a first processor, and the naked-eye 3D display device includes a naked-eye 3D display screen and a second processor; The first processor is used to render the first original scene picture based on the first operation instruction to obtain a first target scene picture, and display the first target scene picture on the VR display screen; The first processor is further configured to push the first target scene image to the second processor upon receiving a first screen projection interaction control event for a first screen projection interaction control; The second processor is used to push the first target scene picture to the naked-eye 3D display screen, and display the first target scene picture through the naked-eye 3D display screen.

4. The interactive display system of multiple display screens according to claim 3, characterized in that: The first processor is further configured to send the first operation instruction to the second processor when a first instruction interaction event for the first instruction control component is detected; The second processor is used to render the first original scene picture based on the first operation instruction to obtain a first target scene picture, and synchronize the first target scene picture to the naked-eye 3D display screen.

5. The interactive display system of multiple display screens according to claim 3, characterized in that: The first processor includes a first data sending component, and the second processor includes a first data receiving component; The first processor is further configured to determine whether the first data sending component includes the first target scene picture, and when it is determined that the data sending component includes the first target scene picture, control the first data sending component to send the first target scene picture to the first data receiving component; The first processor is also used to control the first data sending component to obtain left and right eye display images from the VR display screen when it is determined that the first target scene image is not included in the first data sending component, and to splice the left and right eye display images to obtain the first target scene image, and then send the first target scene image to the first data receiving component.

6. The interactive display system of multiple display screens according to claim 5, characterized in that: The first processor is further used to perform inverse distortion on a first picture rendering result corresponding to the first original scene picture to obtain a first target scene picture; When it is determined that the first data sending component does not include the first target scene picture, the first processor is further used to perform forward distortion on the picture splicing result corresponding to the left and right eye display pictures to obtain the first target scene picture.

7. The interactive display system of multiple display screens according to claim 6, characterized in that: In the process of reversely distorting the first picture rendering result, it is achieved in the following manner: Splitting the first picture rendering result into left-eye and right-eye views to obtain a first left-eye picture rendering result and a first right-eye picture rendering result; Determine a first left eye center point position of a first left eye picture rendering result and a first right eye center point position of a first right eye picture rendering result, and determine a first normal image center point distance according to the first left eye center point position and the first right eye center point position; The center point distance of the first distorted image is determined according to the center point distance of the first normal image, and the picture rendering result is reversely distorted according to the center point distance of the first distorted image to obtain the first target scene picture.

8. The interactive display system of multiple display screens according to claim 6, characterized in that: In the process of performing forward distortion on the image stitching result, it is achieved in the following manner: Determine a second left eye center point position of a second left eye display picture in the picture stitching result, and a second right eye center point position of a second right eye display picture in the picture stitching result, and determine a second distorted image center point distance according to the second left eye center point position and the second right eye center point position; The center point distance of the second normal image is determined according to the center point distance of the second distorted image, and the picture stitching result is forward distorted according to the center point distance of the second normal image to obtain the first target scene picture.

9. The interactive display system of multiple display screens according to claim 4, characterized in that: The first processor also includes a first virtual camera and a first network communication component; the second processor also includes a second virtual camera and a second network communication component; The first virtual camera is used to obtain current device posture data in the device posture operation instruction, and when a first instruction interaction event for the first instruction control component is detected, the current device posture data is sent to the second network communication component via the first network communication component; The second network communication component is used to synchronize the received current device posture data to the second processor; The second processor is used to update the current camera posture of the second virtual camera according to the current device posture data, and render the first original scene picture according to the updated camera posture data to obtain the first target scene picture.

10. The interactive display system of multiple display screens according to claim 9, characterized in that: The first virtual camera is further used to obtain first current lens posture data in the first user operation instruction, and send the first current lens posture data to the second network communication component via the first network communication component; The second network communication component is used to synchronize the received first current lens posture data to the second processor; The second processor is used to update the current camera posture of the second virtual camera according to the first current lens posture data, and render the first original scene picture according to the updated camera posture to obtain the first target scene picture.

11. The interactive display system of multiple display screens according to claim 10, characterized in that: The first current lens posture data includes a unique object identifier of the target virtual object and a first virtual object viewing angle of the target virtual object after executing the first user operation instruction.

12. The interactive display system of multiple display screens according to claim 9, characterized in that: The second processor is further used to: determine a first off-screen scene picture and a first on-screen scene picture included in the first target scene picture, and determine a first picture ratio occupied by the first off-screen scene picture and a second picture ratio occupied by the first on-screen scene picture; The camera type of the second virtual camera is determined according to the first screen ratio and the second screen ratio; wherein the camera type includes at least one of a converging camera, a parallel optical axis camera, and an off-axis camera.

13. The interactive display system of multiple display screens according to claim 12, characterized in that: If the first aspect ratio is smaller than the second aspect ratio, the camera type is determined to be a convergent camera or a parallel optical axis camera; if the first aspect ratio is larger than the second aspect ratio, the camera type is determined to be an off-axis camera.

14. The interactive display system of multiple display screens according to claim 3, characterized in that: The second processor is further used to: obtain first current device posture data of the VR display device at the current moment and first historical device posture data of the previous moment corresponding to the current moment, and determine a first distance difference between the first current device posture data and the first historical device posture data; When it is determined that the first distance difference is less than or equal to a preset distance threshold, plane linear interpolation is performed on the first current device posture data according to the first historical device posture data to obtain a first device posture interpolation result, and the first original scene picture is rendered based on the first device posture interpolation result to obtain a first target scene picture; When it is determined that the first distance difference is greater than a preset distance threshold, the first original scene picture is rendered based on the first current device posture data to obtain a first target scene picture.

15. The interactive display system of multiple display screens according to claim 3, characterized in that: The second processor is further used to: obtain first current lens posture data of the target virtual object at the current moment and first historical lens posture data of the previous moment corresponding to the current moment, and determine whether the posture of the target virtual object rotates according to the first current lens posture data and the first historical lens posture data; When it is determined that the posture of the target virtual object rotates, spherical linear interpolation is performed on the first current lens posture data according to the first historical lens posture data to obtain a first lens posture interpolation result, and a first original scene picture is rendered based on the first lens posture interpolation result to obtain a first target scene picture; When it is determined that the posture of the target virtual object remains unchanged, the first original scene picture is rendered according to the first current lens posture data to obtain a first target scene picture.

16. An interactive display method for multiple display screens, characterized in that: include: In response to a first screen projection interaction control event for a first screen projection interaction control, displaying a first target scene picture in the VR display device to a naked-eye 3D display device, and displaying the first target scene picture; or In response to a first instruction interaction event for a first instruction control component, synchronizing the first operation instruction to the naked-eye 3D display device, and displaying a first target scene image rendered according to the first operation instruction; or Receiving a second target scene picture sent by the naked-eye 3D display device when a second projection screen interaction control event for a second projection screen interaction control is detected, and displaying the second target scene picture; or A second operation instruction sent by the naked-eye 3D display device when a second instruction interaction event for a second instruction control component is detected is received, and a second target scene picture rendered according to the second operation instruction is displayed.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the interactive display method for multiple display screens described in claim 16 is implemented.

18. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the interactive display method of multiple display screens as claimed in claim 16 by executing the executable instructions.

Images