Interactive control method, interactive display method, virtual reality device, and storage medium

Abstract

An interactive control method, an interactive display method, a virtual reality device, and a storage medium. The interactive control method comprises: in response to detecting a trigger operation for a target application, sending a corresponding trigger control message to a target device, so that the target device triggers start of the target application, and sends target interface data corresponding to the target application to a virtual reality device; in response to receiving the target interface data, rendering and displaying the target interface data in a three-dimensional space of the virtual reality device on the basis of a first data transmission channel to form a target interface, wherein the first data transmission channel is used for establishing an association relationship between the three-dimensional space of the virtual reality device and a display interface of the target application; and in response to an interaction operation on the target interface, sending a corresponding operation control message to the target device so as to control the target application in the target device to perform a corresponding operation response. According to the present disclosure, interactive control for external applications can be achieved without the need for the user of the virtual reality device to interrupt the three-dimensional scene, ensuring immersive user experience.

Description

Interaction control method, interaction display method, virtual reality device and storage medium

[0001] Cross-reference to related applications

[0002] The present disclosure claims priority to the application file with the application number of 202410288695.1, the application date of 2024-03-13, and the application name of Interaction control method, interaction display method, virtual reality device and storage medium. The entire contents of the Chinese patent application are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of virtual reality, and particularly relates to an interaction control method, an interaction display method, a virtual reality device and a storage medium. BACKGROUND

[0004] Virtual reality technology is widely promoted and applied because it can bring users an immersive experience. At present, virtual reality devices can only control three-dimensional applications installed on the device. When external applications need to be controlled, the virtual scene needs to be exited to control the external applications, which seriously affects the immersive experience of users.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] According to a first aspect of the present disclosure, an interaction control method of a virtual reality device is provided, the method comprising:

[0007] in response to detecting a trigger operation for a target application, sending a corresponding trigger control message to a target device to trigger the target device to start the target application, and sending target interface data corresponding to the target application to the virtual reality device;

[0008] in response to receiving the target interface data, rendering and displaying the target interface data in a three-dimensional space of the virtual reality device based on a first data transmission channel to form a target interface; the first data transmission channel is used to establish an association relationship between the three-dimensional space of the virtual reality device and the display interface of the target application;

[0009] in response to an interaction operation on the target interface, sending a corresponding operation control message to the target device to control the target application in the target device to perform a corresponding operation response.

[0010] Optionally, the method further comprises: in response to a triggering operation of the target application corresponding control, displaying at least one candidate device corresponding to the target application, the at least one candidate device being a device on which the target application is installed; in response to a selection operation of the at least one candidate device, determining the selected candidate device as the target device, and generating the triggering control message of the target application for the target device.

[0011] Optionally, the method further comprises: in response to detecting a triggering operation of the target application, creating a first texture for image rendering in a three-dimensional space; sending a first texture corresponding identifier to the target device, so that the target device creates a corresponding target surface texture based on the first texture corresponding identifier, creates a corresponding target surface object based on the target surface texture, creates a corresponding first virtual screen based on the target surface object, and takes the first virtual screen as a display interface of the target application to form the first data transmission channel.

[0012] Rendering and displaying the target interface data in the three-dimensional space of the virtual reality device based on the first data transmission channel comprises: rendering the target interface data to the first texture.

[0013] Optionally, rendering the target interface data to the first texture comprises:

[0014] Constructing a target model in a three-dimensional space, the target model being one of a curved surface model, a polygon model, and a spherical model.

[0015] According to the target interface data, performing texture mapping on the target model to render the target interface data to a first texture, the first texture being bound to the target model.

[0016] Optionally, the method further comprises: sending display parameter configuration information to the target device, so that the target device configures target display parameters of the target application to bind the target application to the first virtual screen.

[0017] Optionally, sending a corresponding operation control instruction to the target device to control the target application in the target device to perform a corresponding operation response comprises:

[0018] In response to an interactive operation on the target interface, creating a corresponding simulated triggering event; the simulated triggering event comprising first interaction position information for indicating an interaction point of the interactive operation.

[0019] sending the simulated trigger event to the target device, so that a forwarding listening service in the target device forwards the simulated trigger event to the first virtual screen, so that the first virtual screen responds to the simulated trigger event based on the first interaction position information.

[0020] Optionally, the method further comprises: receiving response interface data sent by the target device, and rendering and displaying the response interface data in the three-dimensional space; the response interface data is generated based on second interaction position information, the second interaction position information is obtained by mapping the interaction point to a corresponding mapping position in the first virtual screen, and the second interaction position information is determined according to the boundary information of the first virtual screen and the first interaction position information.

[0021] Optionally, the method further comprises:

[0022] determining the first coordinates of each vertex in the target polygon corresponding to the target interface and the three-dimensional coordinates of the interaction point;

[0023] determining the distances between the interaction point and each side of the target polygon according to the first coordinates of each vertex and the three-dimensional coordinates of the interaction point;

[0024] determining the proportions of each distance in the corresponding side according to each distance and the boundary length of each side of the target polygon, taking the proportions as the first interaction position information, and causing the target device to determine the second interaction position information according to each proportion.

[0025] Optionally, before determining the first coordinates of each vertex in the target polygon, the method further comprises: in response to the target interface not being a plane, performing a flattening process on the target interface to obtain a corresponding target polygon.

[0026] Optionally, the method further comprises: based on the first data transmission channel, re-rendering the first texture by using the response interface data to update the target interface in the three-dimensional space;

[0027] Alternatively, based on a second data transmission channel, rendering a second texture by using the response interface data to display different interfaces of the target application in different regions of the three-dimensional space; the second data transmission channel is used to establish an association relationship between a second virtual screen of the target device and the second texture, and the second virtual screen is used to display the response interface data in the target device.

[0028] According to a second aspect of the present disclosure, an interaction display method of a virtual reality device is provided, comprising:

[0029] In response to detecting a starting operation for a target application, a starting interface of the target application is rendered and displayed in a target region of a three-dimensional space of the virtual reality device by constructing a first data transmission channel;

[0030] In response to detecting a triggering operation for a first control of the starting interface, corresponding first interface data is rendered and displayed in a first region of the three-dimensional space by constructing a second data transmission channel;

[0031] In response to detecting a triggering operation for a second control of the starting interface, corresponding second interface data is rendered and displayed in a second region of the three-dimensional space by constructing a third data transmission channel;

[0032] The target application is a two-dimensional application, the target region, the first region, and the second region are different regions, the first data transmission channel is used to establish an association relationship between the starting interface and the target region, the second data transmission channel is used to establish an association relationship between a display interface corresponding to the first control and the first region, and the third data transmission channel is used to establish an association relationship between a display interface corresponding to the second control and the second region.

[0033] According to a third aspect of the present disclosure, a virtual reality device is provided, comprising:

[0034] A sending module is configured to, in response to detecting a triggering operation for a target application, send a corresponding triggering control packet to a target device, so that the target device triggers starting of the target application and sends target interface data corresponding to the target application to the virtual reality device.

[0035] A rendering module is configured to, in response to receiving the target interface data, render and display the target interface data in a three-dimensional space of the virtual reality device based on a first data transmission channel, the first data transmission channel being used to establish an association relationship between the three-dimensional space of the virtual reality device and a display interface of the target application.

[0036] An interaction control module is configured to, in response to an interaction operation on a target application interface in the three-dimensional space, send a corresponding control instruction to the target device, so as to control the target application in the target device to perform a corresponding operation response.

[0037] According to a fourth aspect of the present disclosure, a virtual reality device is provided, comprising:

[0038] A first rendering module is configured to, in response to detecting a starting operation for a target application, render and display a starting interface of the target application in a target region of a three-dimensional space of the virtual reality device by constructing a first data transmission channel.

[0039] The second rendering module is configured to, in response to detecting a triggering operation on the first control of the start interface, render and display corresponding first interface data to a first region of the three-dimensional space by constructing a second data transmission channel.

[0040] The third rendering module is configured to, in response to detecting a triggering operation on the second control of the start interface, render and display corresponding second interface data to a second region of the three-dimensional space by constructing a third data transmission channel.

[0041] The target application is a two-dimensional application, the target region, the first region and the second region are different regions, the first data transmission channel is configured to establish an association between the start interface and the target region, the second data transmission channel is configured to establish an association between the display interface corresponding to the first control and the first region, and the third data transmission channel is configured to establish an association between the display interface corresponding to the second control and the second region.

[0042] According to a fifth aspect of the present disclosure, there is provided an interactive control system, comprising at least one of the virtual reality devices in the above embodiments.

[0043] According to a sixth aspect of the present disclosure, there is provided a computer readable medium having stored thereon a computer program, which when executed by a processor, implements the method in the above embodiments.

[0044] According to a seventh aspect of the present disclosure, there is provided an electronic device, comprising: one or more processors; and a storage device storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method in the above embodiments.

[0045] According to an eighth aspect of the present disclosure, there is provided a computer program product comprising a computer program, which when executed by a processor, implements the method of any of the above embodiments.

[0046] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0048] Fig. 1 schematically shows one of flow diagrams of an interaction control method of a virtual reality device according to an embodiment of the present disclosure.

[0049] Fig. 2 schematically shows a diagram of mapping of a target application of a target device to a three-dimensional space of a virtual reality device according to an embodiment of the present disclosure.

[0050] Fig. 3 schematically shows a flow diagram of a construction process of a first data transmission channel according to an embodiment of the present disclosure.

[0051] Fig. 4 schematically shows a flow diagram of a process of responding to an interaction operation in a three-dimensional space in a target device according to an embodiment of the present disclosure.

[0052] Fig. 5 schematically shows a diagram of determining a relative position of an interaction point in a three-dimensional space and mapping the point in a virtual screen according to an embodiment of the present disclosure.

[0053] Fig. 6 schematically shows a diagram of a mapping relationship between an interaction point in a three-dimensional space and a corresponding position in a virtual screen according to an embodiment of the present disclosure.

[0054] Fig. 7 schematically shows a flow diagram of a responding process of a target application to an interaction operation in a three-dimensional space according to an embodiment of the present disclosure.

[0055] Fig. 8 schematically shows a diagram of displaying a plurality of display interfaces of a target application in sub-regions of a three-dimensional space according to an embodiment of the present disclosure.

[0056] Fig. 9 schematically shows another of flow diagrams of an interaction control method of a virtual reality device according to an embodiment of the present disclosure.

[0057] Fig. 10 schematically shows a flow diagram of an interaction display method of a virtual reality device according to an embodiment of the present disclosure.

[0058] Fig. 11 schematically shows a structural block diagram of a virtual reality device according to an embodiment of the present disclosure.

[0059] Fig. 12 schematically shows a structural block diagram of a virtual reality device according to another embodiment of the present disclosure.

[0060] Fig. 13 schematically shows a module diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] Example implementations that can be described in the detailed description section of this disclosure are not meant to be limiting in terms of the scope of what can be claimed as examples can be practiced with combinations of features that are described in this detailed description section. Furthermore, example implementations can be practiced in various ways, and the terminology used in this description is for the purpose of describing the examples and is not intended to limit the scope of the examples.

[0062] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the

[0063] The block diagrams in the drawings show only the functional entities and do not necessarily imply a physical structure for the entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0064] The flow diagrams shown in the drawings are merely examples and do not necessarily include all of the steps and / or operations, nor do the steps and / or operations necessarily need to be performed in the order shown. For example, some operations can be performed in a different order, some operations can be combined or partially combined, and some operations can be skipped or omitted.

[0065] Currently, for a user of a virtual reality device, additional development of an application is needed to adapt the application to its three-dimensional environment. During the use of the virtual reality device by the user, there is often a need to handle other urgent matters outside, at which time the user needs to pause or exit the three-dimensional environment, which will seriously affect the immersive experience of the user and also limit the popularization and application of the virtual reality device.

[0066] Based on the above problems, the present disclosure provides an interactive control method of a virtual reality device.

[0067] In an embodiment of the present disclosure, an interactive control method of a virtual reality device is provided. FIG. 1 shows a flowchart of the interactive control method of the virtual reality device. As shown in FIG. 1, the interactive control method of the virtual reality device can be applied to a virtual reality device. The method at least includes the following steps:

[0068] In step S110, in response to detecting a trigger operation for a target application, a corresponding trigger control packet is sent to a target device to trigger the target device to start the target application, and target interface data corresponding to the target application is sent to the virtual reality device.

[0069] In step S120, in response to receiving the target interface data, the target interface data is rendered and displayed in the three-dimensional space of the virtual reality device based on a first data transmission channel to form a target interface; the first data transmission channel is used to establish an association between the three-dimensional space of the virtual reality device and the display interface of the target application.

[0070] In step S130, in response to the interactive operation on the target interface, a corresponding operation control message is sent to the target device to control the target application in the target device to perform a corresponding operation response.

[0071] The interactive control method of the virtual reality device according to the embodiments of the present disclosure can send a trigger control message for a target application to a target device through a trigger operation of a virtual reality device to achieve start control of the target application; and render and display a display interface of the target application in a three-dimensional space of the virtual reality device based on a first data transmission channel, and achieve interactive control of the target application in the target device through interactive operation in the three-dimensional space, so that a user of the virtual reality device can achieve interactive control of an external application on the target device without interrupting the three-dimensional scene, thereby ensuring immersive experience of the user. On the other hand, additional development of the application to adapt to the three-dimensional space can be avoided, and barrier-free interactive control of the virtual reality device on the external application can be achieved, thereby facilitating popularization and application of the virtual reality device and reducing development cost. In addition, the user can always be in the original operating environment without changing the user's operation habit, thereby improving user acceptance of the product.

[0072] In order to make the technical solutions of the present disclosure clearer, the steps of the interactive control method of the virtual reality device are described as follows.

[0073] In step S110, in response to detecting a trigger operation for a target application, a corresponding trigger control message is sent to a target device to trigger the target device to start the target application and send target interface data corresponding to the target application to the virtual reality device.

[0074] In the example embodiment, the target application can be an external application, for example, an external two-dimensional application. A start control or option of each external application can be configured in the virtual reality device, and the start control or option in the virtual reality device only needs to be operable. A device list installed with the target application can be configured in the virtual reality device, and a corresponding trigger control message can be generated by selecting or detecting a device in the device list. The target device can also be determined or selected through communication negotiation between the virtual reality device and each external device or wireless network connection, for example, an external device connected to the same wireless local area network as the virtual reality device can be preferentially selected as the target device, or the target device can be determined according to communication time or communication conditions, for example, a device with the shortest communication time or the highest communication signal strength can be selected as the target device, and the example is not limited in this regard. In the example, the trigger control message can include target device information (device identification, device code, or device identification code, etc.) and target application information (target application identification or code), and can also include texture information (such as texture identification, etc.) or display area information for constructing a data transmission channel, and can also include other transmission required information, and the example is not limited in this regard. After receiving and analyzing the trigger control message, the target device triggers the start of the corresponding target application, and sends the target interface data of the target application to the virtual reality device for rendering and display. The communication between the virtual reality device and the target device can be realized through at least one of the network transmission protocols such as TCP (Transmission Control Protocol), UDP (User Datagram Protocol), HTTP (HyperText Transfer Protocol), etc.

[0075] For example, the trigger control message can be generated by the following steps: in response to the trigger operation of the target application corresponding control, at least one candidate device is displayed, and the at least one candidate device is a device installed with the target application; in response to the selection operation of the at least one candidate device, the selected candidate device is determined as the target device, and the trigger control message of the target application of the target device is generated.

[0076] In the example embodiment, the candidate device installed with the target application can be displayed by triggering the operation (e.g., clicking, hovering, double-clicking, etc.) of the corresponding control of the target application in the virtual reality device. In the example, the candidate device can be one or more, and the object list indicating the possible control objects of the current virtual reality device can be pre-configured in the virtual reality device. At least one of the icon, name, and identifier of the candidate device can be displayed. The selected device can be determined as the target device by selecting one or more candidate devices, and the corresponding trigger control message can be generated based on the target device information and the target application information to send the instruction to the target device to start the target application. The example can realize the start control of the target application on the specified device through the device selection in the virtual reality device, and improve the convenience of user operation.

[0077] In the example, when the target application is not installed in the target device, the error information can be returned to the virtual reality device, and the object list can be updated based on the error information to improve the subsequent device matching success rate and matching speed.

[0078] In step S120, the target interface data is rendered and displayed in the three-dimensional space of the virtual reality device based on the first data transmission channel in response to receiving the target interface data.

[0079] In the example embodiment, the first data transmission channel is used to establish an association between the three-dimensional space of the virtual reality device and the display interface of the target application. The target device can construct the first data transmission channel in real time based on the received trigger control message when it is determined that the current device has the target application. The target interface data is rendered and displayed in the corresponding area through the first data transmission channel to form the target interface. The area can be determined by temporarily specifying, or can be determined by the system when the trigger control message is generated, which is not limited in the example.

[0080] For example, the first texture for image rendering in the three-dimensional space can be created in response to detecting the trigger operation for the target application; the first texture corresponding identifier is sent to the target device to make the target device create the corresponding target surface texture based on the first texture corresponding identifier, create the corresponding target surface object based on the target surface texture, create the corresponding first virtual screen based on the target surface object, and use the first virtual screen as the display interface of the target application to form the first data transmission channel.

[0081] In the present exemplary embodiment, after the virtual reality device system is started, a three-dimensional space can be created first, and upon detecting a triggering operation for a target application, a first texture for drawing graphics in the three-dimensional space is created, and the first texture is sent to the target device with a corresponding identifier TextureId, which can be included in a triggering control message or other interaction messages, which is not limited in the present example. Exemplarily, the virtual reality device can create the TextureId by calling OpenGL ES (OpenGL for Embedded System), which is a subset of OpenGL three-dimensional graphics application programming interface API and a graphics interface specification designed by OpenGL for mobile terminals or embedded devices such as mobile phones and tablet computers. For example, a TextureId can be obtained by calling the function GLES20.glGenTextures(), and the type of the TextureId can also be specified, for example, the Texture is of the type GLES11Ext.GL_TEXTURE_EXTERNAL_OES, i.e., an external texture type. After receiving the TextureId sent by the OpenGL ES, the target device can create a SurfaceTexture object (target surface texture) through the TextureId, and then create a Surface object (target surface object) based on the SurfaceTexture object, and create a VirtualDisplay object (first virtual screen) based on the Surface object. The target application can be run in the first virtual screen, and the first virtual screen is a virtual interface for displaying the target application, which is invisible and inoperable to the user. As shown in FIG. 2, the virtual reality device can draw a three-dimensional space 210 by using OpenGL ES, and the target device can open and run the target application in the system based on the triggering control message, and the target application is run in a first virtual screen 220. All image data drawn by the first virtual screen 220 is transmitted to the three-dimensional space 210 in real time through a first data transmission channel, and the data is rendered and displayed in the three-dimensional space 210 to obtain a three-dimensional space display interface of the target application, i.e., a target interface 230. After the above data transmission channel is constructed, the target interface data can be rendered to the first texture, and in this way, the interface data of the target application on the target device can be mapped to the corresponding three-dimensional space of the virtual reality device, and the accurate association of the interface data of different devices can be achieved.

[0082] In some embodiments, the target interface data can be rendered to the first texture by the following steps: constructing a target model in the three-dimensional space, the target model being one of a curved surface model, a polygonal model, and a spherical model; and performing texture mapping on the target model according to the target interface data, so as to render the target interface data to the first texture, the first texture being bound to the target model.

[0083] In the present exemplary embodiment, the target model can be constructed in the three-dimensional space based on the display configuration of the target application, the display configuration can include one of a curved surface display, a polygonal display, and a spherical display, and accordingly, a corresponding curved surface model, polygonal model, or spherical model is constructed. The target model can be constructed by selecting a corresponding model from a model library, the target model is bound to the first texture to render the target interface data to the first texture, and the first texture is used as the texture mapping of the target model, so as to realize the display of the target interface data in various forms such as curved surface, polygon, etc. in the three-dimensional space, improve the adaptation of the corresponding interface of the target application to the three-dimensional space, and improve the user experience.

[0084] Exemplarily, the method further includes: sending a display parameter configuration instruction to the target device, so as to make the target device configure the target display parameter of the target application, and bind the target application to the first virtual screen.

[0085] In the present exemplary embodiment, the virtual reality device can determine the display parameter configuration information corresponding to the first texture while constructing the first texture, and send the display parameter configuration information to the target device. The display parameter configuration information can be used to indicate the binding relationship between the target application and the first virtual screen, and can include the configuration value of the target display parameter of the target application. The target device can configure the target display parameter of the target application to a specified value during the execution of the start process of the target application, or can use the identification information displayId of the first virtual screen as the target display parameter of the target application after the first virtual screen is created. The present example does not limit the binding mode of the target application and the corresponding virtual screen (the first virtual screen). The present example realizes the binding of the target application and the corresponding virtual screen by sending the display parameter configuration instruction to the target device, and based on the binding relationship, indicates the virtual screen as the imaging destination of the interface of the target application, so as to realize the display of the application interface without user awareness.

[0086] For example, as shown in FIG. 3, the construction process of the data transmission channel can include the following steps:

[0087] S301, in response to the system startup of the virtual reality device, a three-dimensional space is constructed by using OpenGL ES, specifically, a three-dimensional space can be constructed by using GLSurfaceView control as a drawing environment in Android.

[0088] S302, in response to detecting a trigger operation for the target application, a TextureId is created in OpenGL ES to draw the target application interface.

[0089] S303, the virtual reality device generates a corresponding trigger control message based on the TextureId, the target application ID and the selected target device, and sends the trigger control message to the target device.

[0090] S304, the target device can create a SurfaceTexture object (new SurfaceTexture) through the TextureId.

[0091] S2305, a Surface object (new Surface) is created based on the SurfaceTexture object.

[0092] S306, a VirtualDisplay object is created based on the Surface object; VirtualDisplay is a virtual screen, and the graphics drawn by the virtual screen are output to the Surface object.

[0093] S307, the VirtualDisplay object is bound to the target application, so that when the user opens the target application, the target application is displayed on the bound VirtualDisplay.

[0094] In step S130, in response to the interactive operation on the target interface, a corresponding operation control message is sent to the target device to control the target application in the target device to respond to the operation.

[0095] In the present exemplary embodiment, the user can interact with the target application on the target interface in the three-dimensional space of the virtual reality device, and the interactive operation can be object clicking using a handle, or operation through a gesture, or other arbitrary forms of interface interactive operation, which are not limited in the present example. The corresponding operation control message can be generated in response to the user's interactive operation, and the operation control message is transmitted to the target device to make the target application in the target device respond to the operation. The operation control message can include the position information of the interactive operation, the target application information and the target device information, and can also include other interactive control information, which are not limited in the present example.

[0096] Exemplarily, the interaction control of the target application can be performed through the following steps:

[0097] In a first step, a corresponding simulation trigger event is created in response to the interaction operation on the target interface; the simulation trigger event includes first interaction position information for indicating an interaction point of the interaction operation;

[0098] In a second step, the simulation trigger event is sent to the target device, so that a forwarding listening service in the target device forwards the listened simulation trigger event to the first virtual screen, so that the first virtual screen responds to the simulation trigger event based on the first interaction position information.

[0099] In the present exemplary embodiment, the virtual reality device can generate a corresponding simulation trigger event in response to the interaction operation on the target interface, the simulation trigger event can include first interaction position information, i.e. position information of the current interaction operation, such as interaction point coordinates, relative position of the interaction point in the target interface, etc., and the simulation trigger event is sent to the target device. The target device can listen to the event through an event forwarding listening service, for example, the event forwarding listening service can listen to parameters such as identification information (displayId) of the virtual screen and the simulation trigger event (MotionEvent), and when the simulation trigger event is listened to, the simulation trigger event is forwarded to the virtual screen corresponding to the displayId, so that the target application running in the virtual screen responds to the event. Exemplarily, as shown in FIG. 4, the target device is an Android system, a custom service EventForwardService can be created in advance in the Android Framework layer, and the service listening parameters can include: displayId and MotionEvent (simulation trigger event). The target device responds to the event based on the MotionEvent and displayId listened by the service and the first interaction position information. For example, the target device can generate an operation event corresponding to the position of the first virtual screen. The present example can enable the user to realize the interaction control of the external application in the three-dimensional space without awareness, and at the same time, realize the quick response of the user operation, and guarantee the user experience.

[0100] In some embodiments, the method further comprises: receiving the response interface data sent by the target device, and rendering and displaying the response interface data in the three-dimensional space; the response interface data is generated based on second interaction position information, the second interaction position information is obtained by mapping the interaction point to a corresponding position in the first virtual screen, and the second interaction position information is determined according to the boundary information of the first virtual screen and the first interaction position information.

[0101] In the example embodiment, the first interaction position information can be the relative position of the interaction point in the target interface, the target device can determine the mapping point (i.e. the second interaction position information) in the first virtual screen based on the first interaction position information and the boundary information of the first virtual screen through position mapping, the target device can generate the corresponding response event based on the mapping position, thereby controlling the target application to respond to the corresponding operation, generating the response interface data, and sending the response interface data to the virtual reality device to render and display the response interface data on the virtual reality device, thereby converting the interface interaction operation in the three-dimensional space to the first virtual screen to realize the user's operation control of the external application in the three-dimensional space of the virtual reality device without awareness.

[0102] For example, when the target interface is a polygon, the relative position of the interaction point in the target interface can be determined by the following steps:

[0103] First, determine the first coordinates of each vertex in the target polygon corresponding to the target interface and the three-dimensional coordinates of the interaction point;

[0104] Second, determine the distances between the interaction point and each side of the target polygon according to the first coordinates of each vertex and the three-dimensional coordinates of the interaction point;

[0105] Third, determine the proportion of each distance in the corresponding side according to the distances and the boundary length of each side of the target polygon, and take the proportion as the first interaction position information, so that the target device determines the second interaction position information according to the proportions.

[0106] In the example embodiment, the first coordinates are three-dimensional coordinates of vertices of the target interface in a three-dimensional space, for example, for a rectangular interface, the first coordinates include coordinates of four vertices of the rectangular interface. When the coordinates of the vertices in the three-dimensional space change, for example, the interface moves, rotates, scales, or the like in the three-dimensional space, the positions of the vertices in the three-dimensional space can be calculated based on the motion parameters. The boundary information refers to position information of the boundary, for example, positions / coordinates of four edges of the rectangular interface. The width-height ratio of the target interface and the first virtual screen in the three-dimensional space can be configured to be certain or the same, and the position of the mapping point can be determined according to the relative position relationship between the interaction point and the boundary of the interface and the boundary information of the first virtual screen. For example, for a rectangular interface, the first distance from the interaction point to the first boundary and the second distance from the interaction point to the second boundary can be determined according to the first coordinates of the vertices and the three-dimensional coordinates of the interaction point; the first boundary is the boundary corresponding to the width direction of the interface, and the second boundary is the boundary corresponding to the height direction of the interface; the ratio of the first distance to the boundary length of the second boundary is taken as the first proportion, and the ratio of the second distance to the boundary length of the first boundary is taken as the second proportion; if the width-height ratio of the target interface and the first virtual screen is the same, the mapping point of the interaction point in the first virtual screen can be determined based on the first proportion and the second proportion.

[0107] For example, as shown in FIG. 5, the coordinates of the four vertices of a rectangular interface in a three-dimensional space can be: point LeftTop (x1, y1, z1), point LeftBottom (x2, y2, z2), point RightTop (x3, y3, z3), and point RightBottom (x4, y4, z4). The coordinates of a user's click position on the interface in the three-dimensional space, i.e., the coordinates of the interface interaction point Click, are (x5, y5, z5). The line connecting point LeftTop and point RightTop forms a first boundary, and the line connecting point LeftTop and point LeftBottom forms a second boundary. The first distance S1 of point Click from the first boundary and the second distance S2 of point Click from the second boundary can be calculated according to the distance formula from a point to a straight line in a three-dimensional space. The length of the first boundary and the length of the second boundary are used to determine the proportion of the first distance S1 to the length of the second boundary (the first proportion, i.e., the height proportion) and the proportion of the second distance S2 to the length of the first boundary (the second proportion, i.e., the width proportion). Finally, the specific position PointVD of point Click in the virtual screen is determined according to the boundary mapping proportion relationship between the three-dimensional space corresponding interface and the virtual screen, i.e., the width ratio and the height ratio of the two. The boundary mapping proportion relationship between the three-dimensional space corresponding interface and the virtual screen can be 1:1 mapping or other proportion mapping, which is not limited in this example, but the width-height ratio is the same in the mapping process, which can ensure the display effect. In this example, the relative position of the interface interaction point is converted into a boundary proportion to realize the conversion of the relative position of the three-dimensional space to the relative position of the virtual screen in the two-dimensional space, and then realize the accurate conversion of the position from the three-dimensional space to the two-dimensional space, thereby ensuring the correct response of the user operation.

[0108] For example, as shown in FIG. 6, an external application interface is rendered in a three-dimensional space constructed by OpenGL ES, such as the calculator application shown in the figure. The virtual screen in the figure is the actual display position of the external application, which is invisible and inoperable to the user. As shown by the dashed arrow in the figure, the coordinates of the four vertices of the application interface in the three-dimensional space are mapped to the virtual screen through coordinate conversion, and then the user's click position, i.e., the position near "3" (interaction point), is mapped to point A in the virtual screen. An analog trigger event corresponding to point A is generated to respond to the user's interaction behavior.

[0109] In some embodiments, when the target interface is not a plane, for example, the target interface is a curved surface, the target interface can be flattened before determining the first coordinates of the vertices of the target polygon to obtain the corresponding target polygon.

[0110] In the example embodiment, the target interface corresponding texture map can be flattened based on the target model based on the forming mechanism of the target interface, and a texture map plane, i.e., a target polygon (such as a rectangular plane) is obtained. For example, the flattening process can be performed with a specified point in the target interface as a fixed point, for example, the flattening process can be performed with the center point of the interface as a fixed point, and the fixed point is located on the flattened plane.

[0111] As shown in FIG. 7, the three-dimensional space interaction control process applied in the present disclosure can include the following steps:

[0112] S701, the system of the target device is started.

[0113] S702, the target device creates an event forwarding listening service at the Framework layer of the system, which is used to listen to the simulated trigger events sent from the virtual reality device.

[0114] S703, the user performs an interactive operation on the target interface in the three-dimensional space.

[0115] S704, the system of the virtual reality device determines the relative position of the interaction point in the three-dimensional space (first interaction position information).

[0116] S705, the virtual reality device creates a simulated trigger event based on the first interaction position information, and sends the simulated trigger event to the target device. The simulated trigger event can include the event type and the click position.

[0117] S706, the event forwarding listening service of the target device listens to the simulated trigger event, and forwards the event to the first virtual screen.

[0118] S707, the first virtual screen forwards the event to the target application running in it.

[0119] S708, the target application determines the relative position of the interaction point in the first virtual screen according to the first interaction position information (second interaction position information), and responds to the simulated trigger event.

[0120] The above example embodiment of FIG. 7 realizes accurate and timely response of the system to the user's interactive operation in the three-dimensional space through the conversion of the interaction point position from the three-dimensional space to the two-dimensional space and the listening of the event forwarding listening service to the simulated trigger event, and realizes the user's side non-perception operation.

[0121] In some embodiments, the first texture can be re-rendered based on the response interface data based on the first data transmission channel to update the target interface in the three-dimensional space.

[0122] In the example embodiment, the first texture can be re-rendered using the response interface data to update the target interface, i.e., the display interface of the target application is updated synchronously with the target interface in the three-dimensional space, and the interface synchronization of the target application can be achieved without additional information interaction, transmission and processing, which can save resources and improve operation response speed. In the example, the interface rendering of the target application and the three-dimensional space are the same texture or region.

[0123] Alternatively, in some other embodiments, based on the second data transmission channel, the second texture is rendered using the response interface data to display different interfaces of the target application in different regions of the three-dimensional space; the second data transmission channel is used to establish an association between the second virtual screen of the target device and the second texture, and the second virtual screen is used to display the response interface data in the target device.

[0124] In the example embodiment, based on the user's interaction operation in the three-dimensional space, different data transmission channels can be constructed for the display interfaces corresponding to each interaction operation between the target device and the three-dimensional space of the virtual reality device, so that different display interfaces of the target application can be displayed in different regions of the three-dimensional space. The construction process of each data transmission channel can refer to the construction process of the first data transmission channel, which will not be described here. In the example, the second data transmission channel is different from the first data transmission channel, the second virtual screen is different from the first virtual screen, and the second texture is different from the first texture. Different textures can correspond to different regions of the three-dimensional space, so that the display interfaces of the target application can be displayed in different regions of the three-dimensional space, as shown in FIG. 8, the interface data of the display interface 1, the display interface 2, the display interface 3 and the display interface 4 of the target application are respectively rendered and displayed in the regions A, B, C and D in the three-dimensional space, so that the different display interfaces can be displayed in different regions of the three-dimensional space, which facilitates the user to compare and analyze the content of different interfaces horizontally, and improves the user experience.

[0125] In the example embodiment, the region division of the three-dimensional space can be performed according to the behavior characteristics of the user for each display interface, which refers to the behavior that can reflect the user's usage habits for each display interface. For example, the behavior characteristics can include at least one of the usage frequency, usage time length, stay time length, gaze time length, time interval from the last usage, and the like of the user for each display interface. Different region divisions of the three-dimensional space can be performed based on different user behavior characteristics. For example, a display interface with a long stay time length can be mapped to the middle region of the three-dimensional space, and a display interface with a short stay time length can be mapped to the edge region of the three-dimensional space, so that the interface with high user attention is displayed in the middle region, facilitating the user to capture information. The region division of the three-dimensional space corresponding to the display interface can be performed based on one or more user behavior characteristics, which is not limited in the example.

[0126] For example, as shown in FIG. 9, the interactive control method of the present disclosure can be executed in a system composed of a remote target device and a virtual reality device, and the specific steps are as follows:

[0127] S901, the virtual reality device system is started.

[0128] S902, the virtual reality device initializes the three-dimensional space of the system, that is, constructs the three-dimensional space.

[0129] S903, the virtual reality device detects a triggering operation for the target application, for example, clicking the target application corresponding control in the interface.

[0130] S904, the virtual reality device system calls a three-dimensional graphics drawing tool to create a Texture, and sends the corresponding TextureId to the target device.

[0131] S905, the target device creates a corresponding SurfaceTexture object based on the TextureId.

[0132] S906, the target device creates a corresponding Surface object based on the SurfaceTexture object.

[0133] S907, the target device creates a corresponding virtual screen object (VirtualDisplay) based on the Surface object.

[0134] S908, the target device executes the start of the target application.

[0135] S909, the target device runs the target application in the virtual screen object (VirtualDisplay).

[0136] S910, the target device sends the interface data containing the TextureId to the virtual reality device.

[0137] S911, the virtual reality device displays the interface data based on the TextureId in the three-dimensional space.

[0138] In the example embodiment, the virtual reality device can perform the graphic drawing of the three-dimensional space by calling the OpenGL ES, and the virtual reality device and the target device can be devices with a long physical distance or devices with a short physical distance, and the actual positions of the devices are not limited in the example.

[0139] In other embodiments, referring to FIG. 10, the disclosure also provides an interactive display method of a virtual reality device, which can be applied to a virtual reality device, and the method can include the following steps:

[0140] Step S1010, in response to detecting a starting operation for a target application, rendering and displaying a starting interface of the target application in a target region of a three-dimensional space of the virtual reality device by constructing a first data transmission channel;

[0141] Step S1020, in response to detecting a triggering operation for a first control of the starting interface, rendering and displaying corresponding first interface data to a first region of the three-dimensional space by constructing a second data transmission channel;

[0142] Step S1030, in response to detecting a triggering operation for a second control of the starting interface, rendering and displaying corresponding second interface data to a second region of the three-dimensional space by constructing a third data transmission channel;

[0143] Wherein, the target application is a two-dimensional application, the target region, the first region and the second region are different regions, the first data transmission channel is used to establish an association relationship between the starting interface and the target region, the second data transmission channel is used to establish an association relationship between a display interface corresponding to the first control and the first region, and the third data transmission channel is used to establish an association relationship between a display interface corresponding to the second control and the second region.

[0144] In the example embodiment, the target application can be a two-dimensional application installed in the virtual reality device. The three-dimensional space can be created by starting the specified application, and the user can start the target application by triggering the operation of the control corresponding to the target application. In order to realize the display of different interfaces of the target application in the three-dimensional space, a data transmission channel of the virtual screen of the different interface to the different area of the three-dimensional space can be constructed in the triggering display process of the different interface. The construction process of the data transmission channel can be started by creating an external texture using a three-dimensional graphics drawing tool, and the corresponding virtual screen is constructed based on the external texture, so that the different interfaces are displayed on the corresponding virtual screen. The process of constructing the virtual screen based on the external texture can be referred to the previous embodiment. In the example, the different interfaces of the two-dimensional application can be displayed in the three-dimensional space of the virtual reality device in a region-by-region manner, which facilitates the user to compare the different interfaces of the same application, such as comparing the detail contents of different products in the same shopping application, or comparing the detail contents of different characters or scenes in the same game application, thereby improving the user experience.

[0145] Based on the above-mentioned embodiments, the selected area can be displayed alone or the positions of different areas can be exchanged based on the selection or dragging operation of the user.

[0146] In other embodiments, based on the same inventive concept, different applications can also be displayed in different areas of the three-dimensional space at the same time based on the data transmission channel, so as to realize the comparison of the interface contents of different applications.

[0147] The device embodiments of the present disclosure are introduced below, which can be used to execute the interactive control method of the present disclosure. For details not disclosed in the system embodiments of the present disclosure, please refer to the embodiments of the interactive control method of the present disclosure.

[0148] Referring to FIG. 11, the present disclosure further provides a virtual reality device 1110, which comprises:

[0149] The sending module 1110 is configured to, in response to detecting a triggering operation for a target application, send a corresponding triggering control message to a target device to make the target device trigger to start the target application, and send target interface data corresponding to the target application to the virtual reality device;

[0150] The rendering module 1120 is configured to, in response to receiving the target interface data, render and display the target interface data in the three-dimensional space of the virtual reality device based on a first data transmission channel; the first data transmission channel is used to establish an association relationship between the three-dimensional space of the virtual reality device and the display interface of the target application;

[0151] The control module 1130 is configured to, in response to the interaction operation on the target application interface in the three-dimensional space, send a corresponding control instruction to the target device to control the target application in the target device to perform a corresponding operation response.

[0152] In some embodiments of the present disclosure, based on the foregoing scheme, the device 1100 further includes:

[0153] The message generation module is configured to, in response to a triggering operation on the corresponding control of the target application, display at least one candidate device, the at least one candidate device being a device on which the target application is installed; and in response to a selection operation on the at least one candidate device, determine the selected candidate device as the target device, and generate a triggering control message for the target application of the target device.

[0154] In some embodiments of the present disclosure, based on the foregoing scheme, the rendering module 1120 is configured to:

[0155] In response to detecting the triggering operation on the target application, create a first texture for image rendering in the three-dimensional space;

[0156] Send a first texture corresponding identifier to the target device, so that the target device creates a corresponding target surface texture based on the first texture corresponding identifier, creates a corresponding target surface object based on the target surface texture, creates a corresponding first virtual screen based on the target surface object, and uses the first virtual screen as a display interface of the target application to form a first data transmission channel;

[0157] Render the target interface data to the first texture;

[0158] In some embodiments of the present disclosure, based on the foregoing scheme, the rendering module 1120 is further configured to: construct a target model in the three-dimensional space, the target model being one of a curved surface model, a polygonal model, and a spherical model; and perform texture mapping on the target model according to the target interface data, so as to render the target interface data to the first texture, the first texture being bound to the target model.

[0159] In some embodiments of the present disclosure, based on the foregoing scheme, the sending module 1110 is further configured to:

[0160] Send the display parameter configuration information to the target device, so that the target device configures a target display parameter of the target application, and binds the target application to the first virtual screen.

[0161] In some embodiments of the present disclosure, based on the foregoing scheme, the interaction control module 1130 is further configured to:

[0162] In response to the interaction operation on the target interface, create a corresponding simulated triggering event; the simulated triggering event includes first interaction position information for indicating an interaction point of the interaction operation.

[0163] sending the simulated trigger event to the target device, so that a forwarding listening service in the target device forwards the simulated trigger event listened to to the first virtual screen, so that the first virtual screen responds to the simulated trigger event based on the first interaction position information.

[0164] In some embodiments of the present disclosure, based on the foregoing scheme, the rendering module 1120 is further configured to:

[0165] receive the response interface data sent by the target device, and render and display the response interface data in the three-dimensional space; the response interface data is generated based on the second interaction position information, the second interaction position information is obtained by mapping the interaction point to the corresponding mapping position in the first virtual screen, and the second interaction position information is determined according to the boundary information of the first virtual screen and the first interaction position information.

[0166] In some embodiments of the present disclosure, based on the foregoing scheme, the interaction control module 1130 is further configured to:

[0167] determine the first coordinates of each vertex in the target polygon corresponding to the target interface and the three-dimensional coordinates of the interaction point;

[0168] determine the distances between the interaction point and each side of the target polygon according to the first coordinates of each vertex and the three-dimensional coordinates of the interaction point;

[0169] determine the proportion of each distance in the corresponding side according to the distances and the boundary length of each side of the target polygon, and take the proportion as the first interaction position information, so that the target device determines the second interaction position information according to the proportions.

[0170] In some embodiments of the present disclosure, based on the foregoing scheme, the interaction control module 1130 is further configured to:

[0171] Before determining the first coordinates of each vertex in the target polygon, in response to the target interface not being a plane, the target interface is flattened to obtain the corresponding target polygon.

[0172] In some embodiments of the present disclosure, based on the foregoing scheme, the rendering module is further configured to:

[0173] re-render the first texture based on the first data transmission channel using the response interface data to update the target interface in the three-dimensional space;

[0174] Or, based on the second data transmission channel, render the second texture using the response interface data to display different interfaces of the target application in different regions of the three-dimensional space; the second data transmission channel is used to establish an association relationship between the second virtual screen of the target device and the second texture, and the second virtual screen is used to display the response interface data in the target device.

[0175] The specific details of the modules / units in the virtual reality device have been described in detail in the corresponding interaction control method, and thus will not be described here again.

[0176] Referring to FIG. 12, the disclosure further provides a virtual reality device 1110, comprising:

[0177] The first rendering module 1210 is configured to, in response to detecting a starting operation for a target application, render and display a starting interface of the target application in a target region of a three-dimensional space of the virtual reality device by constructing a first data transmission channel.

[0178] The second rendering module 1220 is configured to, in response to detecting a triggering operation for a first control of the starting interface, render and display corresponding first interface data to a first region of the three-dimensional space by constructing a second data transmission channel.

[0179] The third rendering module 1230 is configured to, in response to detecting a triggering operation for a second control of the starting interface, render and display corresponding second interface data to a second region of the three-dimensional space by constructing a third data transmission channel.

[0180] The target application is a two-dimensional application, the target region, the first region, and the second region are all different regions, the first data transmission channel is configured to establish an association relationship between the starting interface and the target region, the second data transmission channel is configured to establish an association relationship between a display interface corresponding to the first control and the first region, and the third data transmission channel is configured to establish an association relationship between a display interface corresponding to the second control and the second region.

[0181] The specific details of the modules or units in the virtual reality device have been described in detail in the corresponding interaction display method, and thus will not be described here again.

[0182] The disclosure further provides an interaction control system, comprising the virtual reality device of at least one of the foregoing embodiments.

[0183] It should be noted that, although several modules or units of a device for execution are mentioned in the foregoing detailed description, such a division is not mandatory. Indeed, according to the embodiments of the disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into several modules or units.

[0184] In the present exemplary embodiment, an electronic device capable of implementing the above method is also provided.

[0185] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be embodied in a form of entirely hardware, entirely software (including firmware, microcode, etc.), or a combination of hardware and software, which can be generically referred to as "circuitry", "module" or "system".

[0186] The electronic device 1300 according to this embodiment of the present application will be described below with reference to FIG. 13. FIG. 13 shows the electronic device 1300 only as an example, and should not bring any limitation to the functions and usage scope of the embodiments of the present application.

[0187] As shown in FIG. 13, the electronic device 1300 is in the form of a general computing device. The components of the electronic device 1300 can include, but are not limited to, the at least one processing unit 1310 described above, the at least one storage unit 1320 described above, a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310), and a display unit 1340.

[0188] The storage unit stores program codes which can be executed by the processing unit 1310, so that the processing unit 1310 performs the steps according to various exemplary embodiments of the present application described in the interactive control method and the interactive display method described above in the specification. For example, the processing unit 1310 can perform the steps S110 shown in FIG. 1: in response to detecting a trigger operation for a target application, sending a corresponding trigger control message to a target device to make the target device trigger to start the target application, and sending target interface data corresponding to the target application to a virtual reality device; step S120: in response to receiving the target interface data, rendering and displaying the target interface data in a three-dimensional space of the virtual reality device based on a first data transmission channel; the first data transmission channel is used to establish an association relationship between the three-dimensional space of the virtual reality device and the display interface of the target application; step S130: in response to an interactive operation on the target interface, sending a corresponding operation control message to the target device to control the target application in the target device to perform a corresponding operation response.

[0189] The storage unit 1320 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 13201 and / or a cache memory 13202, and can further include a read-only memory (ROM) 13203.

[0190] The storage unit 1320 can also include a program / utility 13204 having a set of programs / modules 13205, each of which performs one or more of the processes to be executed by the processing unit(s) 1310. The programs / utility 13204 can also include, for example, a network environment implementation, which can include each of the examples or some combination thereof, as well as processes for operations associated with the network environment.

[0191] The bus 1330 can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures, and the like.

[0192] The electronic device 1300 can also communicate with one or more external devices such as a keyboard or pointing device, using one or more communication ports 1352. Communication ports 1352 can also enable the electronic device 1300 to communicate with one or more devices that enable a user to interact with the electronic device 1300, and / or one or more devices that enable the electronic device 1300 to communicate with one or more other computing devices. Such communication can be facilitated, for example, by an input / output (I / O) interface 1350. The electronic device 1300 can communicate with one or more networks, such as one or more local area networks (LANs), one or more wide area networks (WANs), and / or one or more public networks, such as the Internet, using a network adapter 1360. As shown, the network adapter 1360 can be communicatively coupled to the other components of the electronic device 1300 via the bus 1330. It should be appreciated that other hardware and / or software components not shown in FIG. 13 can be used in conjunction with the electronic device 1300. For example, the electronic device 1300 can be used in conjunction with a variety of different hardware and / or software components, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0193] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by software in combination with the necessary hardware. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the methods according to the embodiments of the present disclosure.

[0194] In the present exemplary embodiment, a computer readable storage medium is also provided, on which a program product capable of implementing the method described above is stored. In some possible embodiments, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the "Exemplary Method" section above according to various exemplary embodiments of the present application when the program product is run on the terminal device.

[0195] The program product for implementing the method described above according to the embodiments of the present application can take the form of a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited to this, and in the present document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus or device.

[0196] The program product can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0197] The computer readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which readable program codes are borne. Such a propagated data signal can take multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium can also be any readable medium that is not a readable storage medium and that can transmit, propagate or transport program for use by or in connection with an instruction execution system, apparatus or device.

[0198] The program codes contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0199] The program code may be implemented in any of various ways, including procedure-based, narrative-based, object-based, and / or architectural-based versions. In procedure-based implementations, the program code is implemented in a series of isolated procedures, which rely primarily on extrinsic (or global) variables to communicate with one another. In narrative-based implementations, the program code is implemented at various levels by an interpreter or virtual machine. In object-based versions, the program code is implemented as a series of objects, which interact to perform the functionality of the present application. In architectural-based versions, the program code is implemented based on a component model, which organizes functionality into components that are interconnected to achieve the functionality of the present application.

[0200] Further, the above-described diagrams are merely schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended for a limiting purpose. It is readily understood that the processes shown in the above-described diagrams do not indicate or limit the time sequence of the processes. In addition, it is readily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0201] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practice of the present disclosure. The present application is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known, accepted, or customary bounds of the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0202] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated above, and that various modifications and changes in the herein-described structures can be affected without departing from the scope of the present disclosure. The scope of the present disclosure is indicated by the appended claims.

Claims

1. An interactive control method of a virtual reality device, comprising: in response to detecting a trigger operation for a target application, sending a corresponding trigger control message to a target device to make the target device trigger to start the target application and send target interface data corresponding to the target application to the virtual reality device; in response to receiving the target interface data, rendering and displaying the target interface data based on a first data transmission channel in a three-dimensional space of the virtual reality device to form a target interface; the first data transmission channel is used to establish an association relationship between the three-dimensional space of the virtual reality device and the display interface of the target application; in response to an interactive operation on the target interface, sending a corresponding operation control message to the target device to control the target application in the target device to perform a corresponding operation response.

2. The method of claim 1, wherein, The method further comprises: in response to a trigger operation on a corresponding control of the target application, displaying at least one candidate device, the at least one candidate device being a device installed with the target application; in response to a selection operation on at least one of the candidate devices, determining the selected candidate device as the target device, and generating the trigger control message for the target application of the target device.

3. The method of claim 1, wherein, The method further comprises: in response to detecting a trigger operation for a target application, creating a first texture for image rendering in a three-dimensional space; sending a corresponding identifier of the first texture to the target device to make the target device create a corresponding target surface texture based on the corresponding identifier of the first texture, create a corresponding target surface object based on the target surface texture, create a corresponding first virtual screen based on the target surface object, and use the first virtual screen as a display interface of the target application to form the first data transmission channel; rendering and displaying the target interface data in the three-dimensional space of the virtual reality device based on the first data transmission channel, comprising: rendering the target interface data to the first texture.

4. The method of claim 3, wherein, rendering the target interface data to the first texture, comprising: constructing a target model in the three-dimensional space, the target model being one of a curved surface model, a polygonal model, and a spherical model; according to the target interface data, performing texture mapping on the target model to render the target interface data to the first texture, the first texture being bound to the target model.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: sending display parameter configuration information to the target device to make the target device configure target display parameters of the target application to bind the target application to the first virtual screen.

6. The method of claim 4, wherein, sending a corresponding operation control instruction to the target device to control the target application in the target device to perform a corresponding operation response, comprising: in response to an interactive operation on the target interface, creating a corresponding simulated trigger event; the simulated trigger event includes first interaction position information for indicating an interaction point of the interactive operation. sending the simulated trigger event to the target device, so that a forwarding listening service in the target device forwards the simulated trigger event to the first virtual screen, so that the first virtual screen responds to the simulated trigger event based on the first interaction position information.

7. The method of claim 6, wherein, The method further comprises: receiving response interface data sent by the target device, and rendering and displaying the response interface data in the three-dimensional space; the response interface data is generated based on second interaction position information, the second interaction position information is obtained by mapping the interaction point to a corresponding mapping position in the first virtual screen, and the second interaction position information is determined according to the boundary information of the first virtual screen and the first interaction position information.

8. The method of claim 7, wherein, The method further comprises: determining first coordinates of each vertex in a target polygon corresponding to the target interface and three-dimensional coordinates of the interaction point; determining distances between the interaction point and each side of the target polygon according to the first coordinates of each vertex and the three-dimensional coordinates of the interaction point; determining a proportion of each distance in the corresponding side according to each distance and a boundary length of each side of the target polygon, taking the proportion as the first interaction position information, so that the target device determines the second interaction position information according to each proportion.

9. The method of claim 8, wherein, Before determining the first coordinates of each vertex in the target polygon, the method further comprises: in response to the target interface not being a plane, performing a flattening process on the target interface to obtain a corresponding target polygon.

10. The method of claim 7, wherein, The method further comprises: re-rendering the first texture using the response interface data based on the first data transmission channel to update the target interface in the three-dimensional space; or, rendering a second texture using the response interface data based on a second data transmission channel to display different interfaces of the target application in different regions of the three-dimensional space; the second data transmission channel is used to establish an association relationship between a second virtual screen of the target device and the second texture, and the second virtual screen is used to display the response interface data in the target device.

11. An interaction display method of a virtual reality device, comprising: in response to detecting a starting operation for a target application, rendering and displaying a starting interface of the target application in a target region of a three-dimensional space of the virtual reality device by constructing a first data transmission channel; in response to detecting a trigger operation for a first control of the starting interface, rendering and displaying corresponding first interface data to a first region of the three-dimensional space by constructing a second data transmission channel; in response to detecting a trigger operation for a second control of the starting interface, rendering and displaying corresponding second interface data to a second region of the three-dimensional space by constructing a third data transmission channel; The target application is a two-dimensional application, the target region, the first region and the second region are different regions, the first data transmission channel is used to establish an association between the start interface and the target region, the second data transmission channel is used to establish an association between the display interface corresponding to the first control and the first region, and the third data transmission channel is used to establish an association between the display interface corresponding to the second control and the second region.

12. A virtual reality device, comprising: a sending module configured to, in response to detecting a trigger operation for a target application, send a corresponding trigger control message to a target device to trigger the target device to start the target application and send target application corresponding target interface data to the virtual reality device; a rendering module configured to, in response to receiving the target interface data, render and display the target interface data in a three-dimensional space of the virtual reality device based on a first data transmission channel; the first data transmission channel is used to establish an association between the three-dimensional space of the virtual reality device and the display interface of the target application; an interactive control module configured to, in response to an interactive operation on the target application interface in the three-dimensional space, send a corresponding control instruction to the target device to control the target application in the target device to perform a corresponding operation response.

13. A virtual reality device, comprising: a first rendering module configured to, in response to detecting a start operation for a target application, render and display a start interface of the target application in a target region of a three-dimensional space of the virtual reality device by constructing a first data transmission channel; a second rendering module configured to, in response to detecting a trigger operation on a first control of the start interface, render and display corresponding first interface data to a first region of the three-dimensional space by constructing a second data transmission channel; a third rendering module configured to, in response to detecting a trigger operation on a second control of the start interface, render and display corresponding second interface data to a second region of the three-dimensional space by constructing a third data transmission channel; The target application is a two-dimensional application, the target region, the first region and the second region are different regions, the first data transmission channel is used to establish an association between the start interface and the target region, the second data transmission channel is used to establish an association between the display interface corresponding to the first control and the first region, and the third data transmission channel is used to establish an association between the display interface corresponding to the second control and the second region.

14. An interactive control system comprising: The virtual reality device of claim 12 or 13.

15. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-11.

16. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 11.

Images