Interaction control method, interaction display method, virtual reality equipment and storage medium

Abstract

The invention relates to the technical field of virtual reality, and provides an interaction control method, an interaction display method, virtual reality equipment and a storage medium, and the method comprises the steps: in response to a detected triggering operation for a target application, sending a corresponding triggering control message to target equipment, so that the target equipment triggers and starts the target application, target interface data corresponding to the target application is sent to the virtual reality equipment; in response to the received target interface data, rendering and displaying the target interface data in a three-dimensional space of the virtual reality equipment based on the first data transmission channel to form a target interface; the first data transmission channel is used for establishing an association relationship between a three-dimensional space of the virtual reality equipment and a display interface of the target application; and in response to an interaction operation on the target interface, sending a corresponding operation control message to the target device to control a target application in the target device to perform a corresponding operation response. Under the condition that a virtual reality equipment user does not interrupt a three-dimensional scene, interaction control over the external application can be achieved, and the immersive experience of the user is guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual reality technology, and in particular to an interactive control method, an interactive display method, a virtual reality device, and a storage medium. Background Art

[0002] Virtual reality technology has been widely promoted and applied due to its ability to provide users with an immersive experience. Currently, virtual reality devices can only control the 3D applications installed on the device. To control external applications, users must exit the virtual environment, severely impacting the user's immersive experience.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0004] The present invention aims to provide an interactive control method, an interactive display method and apparatus, a virtual reality device, and a storage medium.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0006] According to a first aspect of the present disclosure, there is provided an interactive control method for a virtual reality device, the method comprising:

[0007] In response to detecting a trigger operation for a target application, sending a corresponding trigger control message to a target device, so that the target device triggers and starts the target application, and sends target interface data corresponding to the target application to the virtual reality device;

[0008] In response to receiving the target interface data, rendering and displaying the target interface data in the three-dimensional space of the virtual reality device based on a first data transmission channel to form a target interface; the first data transmission channel is used to establish an association between the three-dimensional space of the virtual reality device and the display interface of the target application;

[0009] In response to the interactive operation on the target interface, a corresponding operation control message is sent to the target device to control the target application in the target device to perform a corresponding operation response.

[0010] Optionally, the method further includes: in response to a triggering operation on a control corresponding to the target application, displaying at least one corresponding candidate device, wherein the at least one candidate device is a device on which the target application is installed; in response to a selection operation on at least one of the candidate devices, determining that the selected candidate device is the target device, and generating the trigger control message for the target application of the target device.

[0011] Optionally, the method further includes: in response to detecting a trigger operation for the target application, creating a first texture for image drawing in a three-dimensional space; sending the first texture corresponding identifier to a target device, so that the target device creates a corresponding target surface texture based on the first texture corresponding identifier, creates a corresponding target surface object based on the target surface texture, creates a corresponding first virtual screen based on the target surface object, and uses the first virtual screen as a display interface of the target application to form the first data transmission channel;

[0012] Rendering and displaying the target interface data in the three-dimensional space of the virtual reality device based on the first data transmission channel includes: rendering the target interface data to the first texture.

[0013] Optionally, rendering the target interface data to the first texture includes:

[0014] Constructing a target model in a three-dimensional space, wherein the target model is one of a surface model, a polygonal model, and a spherical model;

[0015] According to the target interface data, texture mapping is performed on the target model to render the target interface data to a first texture, and the first texture is bound to the target model.

[0016] Optionally, the method further includes: sending display parameter configuration information to the target device, so that the target device configures target display parameters of the target application, so as to bind the target application to the first virtual screen.

[0017] Optionally, sending a corresponding operation control instruction to the target device to control a target application in the target device to perform a corresponding operation response includes:

[0018] In response to the interactive operation on the target interface, a corresponding simulation trigger event is created; the simulation trigger event includes first interaction position information for indicating an interaction point of the interactive operation;

[0019] The simulation trigger event is sent to the target device, so that the forwarding monitoring service in the target device forwards the monitored simulation trigger event to the first virtual screen, so that the first virtual screen responds to the simulation trigger event based on the first interaction position information.

[0020] Optionally, the method also includes: receiving response interface data sent by the target device, and rendering and displaying the response interface data in three-dimensional space; the response interface data is generated based on second interaction position information, the second interaction position information is mapping the interaction point to a corresponding mapping position in the first virtual screen, and the second interaction position information is determined based on the boundary information of the first virtual screen and the first interaction position information.

[0021] Optionally, the method further includes:

[0022] Determine the first coordinates of each vertex in the target polygon corresponding to the target interface and the three-dimensional coordinates of the interaction point;

[0023] Determine the distance between the interaction point and each side of the target polygon according to the first coordinates of each vertex and the three-dimensional coordinates of the interaction point;

[0024] According to each of the distances and the boundary length of each side of the target polygon, the proportion of each distance in the corresponding side is determined, and the proportion is used as the first interaction position information, so that the target device determines the second interaction position information according to each proportion.

[0025] Optionally, before determining the first coordinates of each vertex in the target polygon, the method further includes: in response to the target interface being not a plane, flattening the target interface to obtain a corresponding target polygon.

[0026] Optionally, the method further includes: re-rendering the first texture using the response interface data based on the first data transmission channel to update the target interface in the three-dimensional space;

[0027] Alternatively, based on a second data transmission channel, the response interface data is used to render a second texture to display different interfaces of the target application in different areas of the three-dimensional space; the second data transmission channel is used to establish an association relationship between the second virtual screen of the target device and the second texture, and the second virtual screen is used to display the response interface data in the target device.

[0028] According to a second aspect of the present disclosure, there is provided an interactive display method for a virtual reality device, comprising:

[0029] In response to detecting a startup operation for a target application, rendering and displaying a startup interface of the target application in a target area of ​​the three-dimensional space of the virtual reality device by establishing a first data transmission channel;

[0030] In response to detecting a triggering operation on a first control of the startup interface, rendering and displaying corresponding first interface data in a first area of ​​the three-dimensional space by constructing a second data transmission channel;

[0031] In response to detecting a triggering operation on a second control of the startup interface, rendering and displaying corresponding second interface data in a second area of ​​the three-dimensional space by constructing a third data transmission channel;

[0032] In which, the target application is a two-dimensional application, the target area, the first area and the second area are all different areas, the first data transmission channel is used to establish an association relationship between the startup interface and the target area, the second data transmission channel is used to establish an association relationship between the display interface corresponding to the first control and the first area, and the third data transmission channel is used to establish an association relationship between the display interface corresponding to the second control and the second area.

[0033] According to a third aspect of the present disclosure, there is provided a virtual reality device, comprising:

[0034] a sending module, configured to, in response to detecting a trigger operation for a target application, send a corresponding trigger control message to a target device, so that the target device triggers and starts the target application, and sends target interface data corresponding to the target application to the virtual reality device;

[0035] a rendering module, configured to, in response to receiving the target interface data, render and display the target interface data in a three-dimensional space of the virtual reality device based on a first data transmission channel; the first data transmission channel being configured to establish an association between the three-dimensional space of the virtual reality device and a display interface of a target application;

[0036] The interactive control module is used to respond to the interactive operation on the target application interface in the three-dimensional space and send a corresponding control instruction to the target device to control the target application in the target device to perform a corresponding operation response.

[0037] According to a fourth aspect of the present disclosure, there is provided a virtual reality device, comprising:

[0038] a first rendering module, configured to, in response to detecting a startup operation for a target application, render and display a startup interface of the target application in a target area of ​​the three-dimensional space of the virtual reality device by establishing a first data transmission channel;

[0039] A second rendering module is configured to, in response to detecting a triggering operation on the first control of the startup interface, render and display the corresponding first interface data in a first area of ​​the three-dimensional space by constructing a second data transmission channel;

[0040] a third rendering module, configured to, in response to detecting a triggering operation on the second control of the startup interface, render and display the corresponding second interface data in the second area of ​​the three-dimensional space by constructing a third data transmission channel;

[0041] In which, the target application is a two-dimensional application, the target area, the first area and the second area are all different areas, the first data transmission channel is used to establish an association relationship between the startup interface and the target area, the second data transmission channel is used to establish an association relationship between the display interface corresponding to the first control and the first area, and the third data transmission channel is used to establish an association relationship between the display interface corresponding to the second control and the second area.

[0042] According to a fifth aspect of the present disclosure, an interactive control system is provided, comprising at least one of the virtual reality devices in the above embodiments.

[0043] According to a sixth aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method in the above embodiment is implemented.

[0044] According to the seventh aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the method as in the above-mentioned embodiment.

[0045] According to an eighth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program product is executed by a processor, the method of any of the above embodiments is implemented.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0048] Figure 1One of the flow charts of the interactive control method of a virtual reality device according to an embodiment of the present disclosure is schematically shown.

[0049] Figure 2 The diagram schematically shows mapping a target application of a target device to a three-dimensional space of a virtual reality device according to an embodiment of the present disclosure.

[0050] Figure 3 The figure schematically shows a flow chart of a process of constructing a first data transmission channel according to an embodiment of the present disclosure.

[0051] Figure 4 The diagram schematically shows a process diagram of responding to interactive operations in a three-dimensional space in a target device according to an embodiment of the present disclosure.

[0052] Figure 5 The diagram schematically shows the determination of the relative position of an interaction point in a three-dimensional space and its mapping point on a virtual screen according to an embodiment of the present disclosure.

[0053] Figure 6 A schematic diagram schematically illustrates a mapping relationship between interaction points in a three-dimensional space and corresponding positions in a virtual screen according to an embodiment of the present disclosure.

[0054] Figure 7 The following schematically illustrates a process diagram of a target application responding to an interactive operation in a three-dimensional space according to an embodiment of the present disclosure.

[0055] Figure 8 The diagram schematically shows the display of multiple display interfaces of a target application in different regions in a three-dimensional space according to an embodiment of the present disclosure.

[0056] Figure 9 The second flowchart of the interactive control method of a virtual reality device according to an embodiment of the present disclosure is schematically shown.

[0057] Figure 10 The following schematically illustrates a flow chart of an interactive display method for a virtual reality device according to an embodiment of the present disclosure.

[0058] Figure 11 The following schematically shows a structural block diagram of a virtual reality device according to an embodiment of the present disclosure.

[0059] Figure 12 The following schematically shows a structural block diagram of a virtual reality device according to another embodiment of the present disclosure.

[0060] Figure 13 The figure schematically shows a module diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0062] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present disclosure.

[0063] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0064] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0065] Currently, users of virtual reality devices need to undergo additional application development to adapt them to their 3D environments. However, while using a virtual reality device, users often need to attend to urgent external matters, pausing or exiting the 3D environment. This severely impacts the user's immersive experience and limits the widespread adoption of virtual reality devices.

[0066] In view of the above problems, the present disclosure provides an interactive control method for a virtual reality device.

[0067] In one embodiment of the present disclosure, an interactive control method for a virtual reality device is proposed. Figure 1 A flow chart of an interactive control method for a virtual reality device is shown. Figure 1 As shown, the interactive control method of the virtual reality device can be applied to the virtual reality device. The method includes at least the following steps:

[0068] Step S110, in response to detecting a trigger operation for a target application, sending a corresponding trigger control message to a target device, so that the target device triggers to start the target application, and sends target interface data corresponding to the target application to the virtual reality device.

[0069] Step S120: In response to receiving the target interface data, the target interface data is rendered and displayed in the three-dimensional space of the virtual reality device based on a first data transmission channel to form a target interface; the first data transmission channel is used to establish an association between the three-dimensional space of the virtual reality device and the display interface of the target application.

[0070] Step S130 : In response to the interactive operation on the target interface, a corresponding operation control message is sent to the target device to control the target application in the target device to perform a corresponding operation response.

[0071] The interactive control method for a virtual reality device in the disclosed embodiment can achieve startup control of the target application by sending a trigger control message for the target application to the target device through a trigger operation on the virtual reality device side; then, based on a first data transmission channel, the display interface of the target application is rendered and displayed in the three-dimensional space of the virtual reality device. Through interactive operations in the three-dimensional space, interactive control of the target application on the target device is achieved, allowing the user of the virtual reality device to achieve interactive control of external applications on the target device without interrupting the three-dimensional scene, thereby ensuring the user's immersive experience. On the other hand, it can avoid additional development of applications to adapt to the three-dimensional space, and can achieve barrier-free interactive control of external applications by the virtual reality device, facilitating the promotion and application of virtual reality devices and reducing development costs. In addition, it can keep users in their original operating environment at all times without changing their operating habits, thereby improving user acceptance of the product.

[0072] In order to make the technical solution of the present disclosure clearer, the steps of the interactive control method of the virtual reality device are described below.

[0073] In step S110 , in response to detecting a trigger operation for a target application, a corresponding trigger control message is sent to a target device, so that the target device triggers the start of the target application and sends target interface data corresponding to the target application to the virtual reality device.

[0074] In this exemplary embodiment, the target application can be an external application, such as an external two-dimensional application. Startup controls or options for various external applications can be configured in the virtual reality device. The virtual reality device only needs to ensure that the startup controls or options are operable. A list of devices with target applications installed can be configured in the virtual reality device, so that a corresponding trigger control message can be generated by selecting or detecting devices in the device list. The target device can also be determined or selected through communication negotiation between the virtual reality device and each external device, or through wireless network connectivity. For example, external devices connected to the same wireless local area network as the virtual reality device can be prioritized as target devices. Target devices can also be determined based on communication time or communication conditions, such as the device with the shortest communication time or the highest communication signal strength, though this example is not limited to this. In this example, the trigger control message can include target device information (device identification, device code, or device identification code, etc.) and target application information (target application identification or code). It can also include texture information (such as a texture identification, etc.) or display area information used to establish a data transmission channel. It can also include other information required for transmission, though this example is not limited to this. After receiving and parsing the trigger control message, the target device triggers the launch of the corresponding target application and sends the target interface data corresponding to the target application to the virtual reality device for rendering and display. Communication between the virtual reality device and the target device can be achieved through at least one network transmission protocol such as TCP (Transmission Control Protocol), UDP (User Datagram Protocol), HTTP (Hypertext Transfer Protocol), etc.

[0075] Exemplarily, a trigger control message can be generated through the following steps: in response to a trigger operation on a control corresponding to a target application, displaying at least one corresponding candidate device, where at least one candidate device is a device on which the target application is installed; in response to a selection operation on at least one candidate device, determining that the selected candidate device is a target device, and generating a trigger control message for the target application of the target device.

[0076] In this exemplary embodiment, the candidate devices on which the target application is installed can be displayed by triggering the corresponding control of the target application in the virtual reality device (such as clicking, floating, double-clicking, etc.). In this example, there can be one or more candidate devices, and an object list can be pre-configured in the virtual reality device. The object list is used to indicate the possible control objects of the current virtual reality device. At least one of the icons, names, and logos of the candidate devices can be displayed. By selecting one or more candidate devices, the corresponding selected device can be determined as the target device, and a corresponding trigger control message can be generated based on the target device information and the target application information to send the instruction to the target device, instructing the target device to start the target application. This example can realize the startup control of the target application on the specified device through device selection in the virtual reality device, thereby improving the convenience of user operation.

[0077] In this example, when the target application is not installed in the target device, an error message may be returned to the virtual reality device. The virtual reality device may update the object list based on the error message to improve the subsequent device matching success rate and matching speed.

[0078] In step S120 , in response to receiving the target interface data, the target interface data is rendered and displayed in the three-dimensional space of the virtual reality device based on the first data transmission channel.

[0079] In this exemplary embodiment, the first data transmission channel is used to establish an association between the three-dimensional space of the virtual reality device and the display interface of the target application. Based on the received trigger control message, the target device can establish the first data transmission channel in real time when it determines that the current device has the target application. The target interface data is rendered and displayed in the corresponding area via the first data transmission channel to form the target interface. This area can be determined by temporary designation or by system allocation when the trigger control message is generated. This example does not limit this.

[0080] Exemplarily, in response to detecting a trigger operation for a target application, a first texture for image drawing in a three-dimensional space can be created; a first texture corresponding identifier is sent to a target device so that the target device creates a corresponding target surface texture based on the first texture corresponding identifier, creates a corresponding target surface object based on the target surface texture, creates a corresponding first virtual screen based on the target surface object, and uses the first virtual screen as a display interface of the target application to form a first data transmission channel.

[0081] In this exemplary embodiment, after the virtual reality device system is started, a three-dimensional space can be created first. When a trigger operation for a target application is detected, a first texture for drawing graphics in the three-dimensional space is created, and the first texture corresponding identifier TextureId is sent to the target device. The TextureId can be included in the trigger control message or in other interactive messages. This example does not limit this. For example, the virtual reality device can create a TextureId by calling OpenGL ES (OpenGL for Embedded System). OpenGL ES is a subset of the OpenGL three-dimensional graphics application program interface API. It is a graphics interface specification designed by OpenGL for mobile terminals or embedded devices such as mobile phones and tablets. For example, a TextureId can be obtained through the GLES20.glGenTextures() function, and the Texture type corresponding to the TextureId can also be specified, such as specifying the Texture as the GLES11Ext.GL_TEXTURE_EXTERNAL_OES type, that is, an external texture type. After receiving the TextureId sent by OpenGL ES, the target device can create a SurfaceTexture object (target surface texture) through the TextureId, and then create a Surface object (target surface object) based on the SurfaceTexture object, and create a VirtualDisplay object (first virtual screen) based on the Surface object, and can run the target application on the first virtual screen. The first virtual screen is a virtual interface for displaying the target application. The first virtual screen is invisible and inoperable to the user. Figure 2 As shown, a virtual reality device can use OpenGL ES to render a three-dimensional space 210. Based on a trigger control message, the target device opens and runs a target application in the system. This target application runs on a first virtual screen 220. All image data drawn on the first virtual screen 220 is transmitted to the three-dimensional space 210 in real time via a first data transmission channel. The data is then rendered and displayed in the three-dimensional space 210, resulting in the target application's three-dimensional display interface, namely, the target interface 230. After establishing this data transmission channel, the target interface data can be rendered to the first texture. This method allows the target application's interface data on the target device to be mapped to the three-dimensional space corresponding to the virtual reality device, achieving accurate association of interface data across different devices.

[0082] In some embodiments, the target interface data can be rendered to the first texture through the following steps: constructing a target model in three-dimensional space, where the target model is one of a surface model, a polygonal model, and a spherical model; performing texture mapping on the target model according to the target interface data to render the target interface data to the first texture, and the first texture is bound to the target model.

[0083] In this exemplary embodiment, a target model can be constructed in three-dimensional space based on the display configuration for the target application. The display configuration can include one of curved surface display, polygonal display, and spherical display. Accordingly, a corresponding curved surface model, polygonal model, or spherical model is constructed. The target model can be constructed by selecting a corresponding model from a model library, binding the target model to a first texture, rendering the target interface data to the first texture, and using the first texture as a texture map for the target model. This enables the target interface data to be displayed in various forms in three-dimensional space, such as curved surfaces and polygons, improving the compatibility of the target application's corresponding interface with the three-dimensional space and enhancing the user experience.

[0084] Exemplarily, the method further includes: sending a display parameter configuration instruction to the target device, so that the target device configures target display parameters of the target application, so as to bind the target application data to the first virtual screen.

[0085] In this exemplary embodiment, while constructing the first texture, the virtual reality device can determine the display parameter configuration information corresponding to the first texture and send the display parameter configuration information to the target device. A display parameter configuration message can be generated based on the display parameter configuration information and the first texture corresponding identifier and sent to the target device. The display parameter configuration information is used to indicate the binding relationship between the target application and the first virtual screen, and may include the configuration value of the target display parameter of the target application. The target device can configure the target display parameter of the target application to a specified value during the startup process of the target application, or use the identification information displayId of the first virtual screen as the target display parameter of the target application after the first virtual screen is created. This example does not limit the binding method of the target application and the corresponding virtual screen (first virtual screen). This example implements the binding of the target application and the corresponding virtual screen by sending a display parameter configuration instruction to the target device, and indicates the interface imaging destination of the target application as the virtual screen based on the binding relationship, so as to achieve user-unaware application interface display.

[0086] For example, if Figure 3 As shown, the process of constructing a data transmission channel may include the following steps:

[0087] S301, in response to the system startup of the virtual reality device, construct a three-dimensional space using OpenGL ES. Specifically, a GLSurfaceView control can be used as a drawing environment in Android to construct a three-dimensional space.

[0088] S302 : In response to detecting a trigger operation for a target application, creating a TextureId in OpenGL ES to draw the target application interface.

[0089] S303: The virtual reality device generates a corresponding trigger control message based on TextureId, the target application ID, and the selected target device, and sends the trigger control message to the target device.

[0090] S304: The target device may create a SurfaceTexture object (newSurfaceTexture) using the TextureId.

[0091] S2305: Create a Surface object (new Surface) based on the SurfaceTexture object.

[0092] S306 , creating a VirtualDisplay object based on the Surface object; the VirtualDisplay is a virtual screen, and graphics drawn on the virtual screen are output to the Surface object.

[0093] S307: Bind the VirtualDisplay object to the target application so that when the user opens the target application, it is displayed on the bound VirtualDisplay.

[0094] In step S130 , in response to the interactive operation on the target interface, a corresponding operation control message is sent to the target device to control the target application in the target device to perform a corresponding operation response.

[0095] In this exemplary embodiment, the user can perform interactive operations on the target application in the target interface in the three-dimensional space of the virtual reality device. The interactive operation can be to click on the object using a handle, or to perform operations through gestures, or any other form of interface interactive operation, which is not limited in this example. A corresponding operation control message can be generated in response to the user's interactive operation, and the operation control message can be transmitted to the target device so that the target application in the target device responds to the operation. The operation control message can include the location information, target application information and target device information of the interactive operation, and can also include other information required for interactive control, which is not limited in this example.

[0096] Exemplarily, interactive control of the target application can be performed through the following steps:

[0097] The first step is to create a corresponding simulation trigger event in response to an interactive operation on a target interface; the simulation trigger event includes first interaction position information for indicating an interaction point of the interactive operation;

[0098] In the second step, the simulation trigger event is sent to the target device, so that the forwarding monitoring service in the target device forwards the monitored simulation trigger event to the first virtual screen, so that the first virtual screen responds to the simulation trigger event based on the first interaction position information.

[0099] In this exemplary embodiment, the virtual reality device can generate a corresponding simulation trigger event in response to an interactive operation on the target interface. The simulation trigger event can include first interaction position information, that is, the position information of the current interactive operation, such as the coordinates of the interaction point, the relative position of the interaction point in the target interface, etc., and send the simulation trigger event to the target device. The target device can monitor the event through the event forwarding monitoring service. For example, the event forwarding monitoring service can monitor the identification information (displayId) of the virtual screen and the parameters such as the simulation trigger event (MotionEvent). When the simulation trigger event is monitored, the simulation trigger event is forwarded to the virtual screen corresponding to the displayId, so that the target application running in the virtual screen responds to the event. For example, Figure 4 As shown, the target device is an Android system, and a custom service EventForwardService can be created in advance in the Android Framework layer. The service monitoring parameters may include: displayId and MotionEvent (simulated trigger event). The target device responds to the event based on the MotionEvent and displayId monitored by the service and the first interactive position information. For example, the target device can generate an operation event corresponding to the position of the first virtual screen. This example allows users to realize interactive control of external applications in three-dimensional space without perception, and at the same time, achieves rapid response to user operations to ensure user experience.

[0100] In some embodiments, the method also includes: receiving response interface data sent by the target device, and rendering and displaying the response interface data in three-dimensional space; the response interface data is generated based on second interaction position information, and the second interaction position information is mapping the interaction point to the corresponding position in the first virtual screen, and the second interaction position information is determined based on the boundary information of the first virtual screen and the first interaction position information.

[0101] In this example embodiment, the first interaction position information may be the relative position of the interaction point in the target interface. The target device may determine the mapping point (i.e., the second interaction position information) in the first virtual screen through position mapping based on the first interaction position information and the boundary information of the first virtual screen. The target device may generate a corresponding response event based on the mapping position, thereby controlling the target application to respond to the corresponding operation, generate response interface data, and send the response interface data to the virtual reality device to render and display the response interface data on the virtual reality device, thereby converting the interface interaction operation in the three-dimensional space to the first virtual screen, so as to realize user-imperceptible operation control of the external application in the three-dimensional space of the virtual reality device.

[0102] For example, when the target interface is a polygon, the relative position of the interaction point in the target interface can be determined by the following steps:

[0103] The first step is to determine the first coordinates of each vertex in the target polygon corresponding to the target interface and the three-dimensional coordinates of the interaction point;

[0104] The second step is to determine the distance between the interaction point and each side of the target polygon based on the first coordinates of each vertex and the three-dimensional coordinates of the interaction point;

[0105] In the third step, based on each distance and the boundary length of each side of the target polygon, the proportion of each distance in the corresponding side is determined, and the proportion is used as the first interaction position information, so that the target device determines the second interaction position information based on each proportion.

[0106] In this example embodiment, the first coordinate is the three-dimensional coordinate of each vertex of the target interface in the three-dimensional space. For example, for a rectangular interface, the first coordinate includes the coordinates of the four vertices of the rectangular interface. When the vertex coordinates in the three-dimensional space change, for example, the interface moves, rotates, scales, etc. arbitrarily in the three-dimensional space, the position of each vertex in the three-dimensional space can be calculated based on the motion parameters. Boundary information refers to the position information of the boundary, such as the position / coordinates of the four sides of the rectangular interface. The aspect ratio of the target interface in the three-dimensional space and the first virtual screen can be configured to be constant or the same, and the position of the mapping point can be determined based on the relative position relationship between the interaction point and the interface boundary and the boundary information of the first virtual screen. Exemplarily, for a rectangular interface, the first distance from the interaction point to the first boundary and the second distance from the interaction point to the second boundary can be determined based on the first coordinates of each vertex and the three-dimensional coordinates of the interaction point; the first boundary is the boundary corresponding to the width direction of the interface, and the second boundary is the boundary corresponding to the height direction of the interface; the ratio of the first distance to the boundary length of the second boundary is taken as the first ratio, and the ratio of the second distance to the boundary length of the first boundary is taken as the second ratio; if the target interface has the same aspect ratio as the first virtual screen, the mapping point of the interaction point in the first virtual screen can be determined based on the first ratio and the second ratio.

[0107] For example, if Figure 5As shown, the coordinates of the four vertices of the rectangular interface in three-dimensional space are: LeftTop(x1,y1,z1), LeftBottom(x2,y2,z2), RightTop(x3,y3,z3), RightBottom(x4,y4,z4). The coordinates of the user's click position on the corresponding interface in three-dimensional space, that is, the interface interaction point Click, are (x5,y5,z5). Then the edge formed by the line connecting the points LeftTop and RightTop is the first boundary, and the line connecting the points LeftTop and LeftBottom is the second boundary. The edge formed is the second boundary; the first distance S1 from the point Click to the first boundary and the second distance S2 from the point Click to the second boundary can be calculated according to the distance formula from a point to a straight line in three-dimensional space; then, according to the lengths of the first boundary and the second boundary, determine the proportion of the first distance S1 to the length of the second boundary (the first proportion is the height proportion), and the proportion of the second distance S2 to the length of the first boundary (the second proportion is the width proportion); finally, according to the boundary mapping ratio relationship between the application interface corresponding to the three-dimensional space and the virtual screen, that is, the width ratio and height ratio of the two, determine the specific position PointVD of the point Click in the virtual screen. The boundary mapping ratio relationship between the corresponding interface in the three-dimensional space and the virtual screen can be a 1:1 mapping or other ratio mapping. This example does not limit this, and the aspect ratio in the mapping process is the same. The same aspect ratio can ensure the display effect. This example converts the relative position of the interface interaction point into the boundary ratio to achieve the conversion of the relative position of the three-dimensional space to the relative position of the virtual screen in the two-dimensional space, thereby achieving accurate conversion of the position from the three-dimensional space to the two-dimensional space, and ensuring the correct response to user operations.

[0108] For example, if Figure 6 As shown, an external application interface, such as the calculator application shown in the figure, is rendered in a 3D space constructed using OpenGL ES. The virtual screen in the figure is the actual display location of the external application and is invisible and inoperable to the user. As shown by the dotted arrows in the figure, the coordinates of the four vertices of the application interface in the 3D space are mapped to the virtual screen through coordinate transformation. The user's click location, near "3" (interaction point), is then mapped to point A on the virtual screen. A simulated trigger event corresponding to point A is generated in response to the user's interaction.

[0109] In some embodiments, when the target interface is not a plane, for example, the target interface is a curved surface, before determining the first coordinates of each vertex in the target polygon, the target interface may be flattened to obtain the corresponding target polygon.

[0110] In this example embodiment, based on the formation mechanism of the target interface and the target model, the texture map corresponding to the target interface can be flattened to obtain a texture map plane, i.e., a target polygon (e.g., a rectangular plane). For example, the flattening process can be performed with a specified point in the target interface as a fixed point. For example, the flattening process can be performed with the center point of the interface as a fixed point, and the fixed point is located on the flattened plane.

[0111] like Figure 7 As shown, the three-dimensional space interactive control process of the target application in the present disclosure may include the following steps:

[0112] S701, the system of the target device is started.

[0113] S702: The target device creates an event forwarding monitoring service in the framework layer of the system to monitor the simulation trigger event sent from the virtual reality device.

[0114] S703: The user performs interactive operations on the target interface in the three-dimensional space.

[0115] S704: The system of the virtual reality device determines the relative position of the interaction point in the three-dimensional space (first interaction position information).

[0116] S705: The virtual reality device creates a simulation trigger event based on the first interaction position information and sends the simulation trigger event to the target device. The simulation trigger event may include an event type and a click position.

[0117] S706: The event forwarding monitoring service of the target device monitors the simulation triggering event and forwards the event to the first virtual screen.

[0118] S707: The first virtual screen forwards the event to the target application running therein.

[0119] S708: The target application determines the relative position (second interaction position information) of the interaction point in the first virtual screen according to the first interaction position information, and responds to the simulation trigger event.

[0120] above Figure 7 The solution of the example embodiment realizes accurate and timely response of the system to the user's interactive operations in three-dimensional space by converting the interaction point position from three-dimensional space to two-dimensional space and monitoring the simulated trigger events by the event forwarding monitoring service, thereby realizing non-perceptual operation on the user side.

[0121] In some embodiments, the first texture may be re-rendered using the response interface data based on the first data transmission channel to update the target interface in the three-dimensional space.

[0122] In this exemplary embodiment, the first texture can be re-rendered using the response interface data to update the target interface. That is, the display interface of the target application is updated and synchronized with the target interface in the three-dimensional space. This synchronization of the target application interface can be achieved without the need for additional information exchange, transmission, and processing, which can save resources and improve the operation response speed. In this example, the interface of the target application is rendered in the same texture or area as the three-dimensional space.

[0123] Alternatively, in other embodiments, based on the second data transmission channel, the second texture is rendered using the response interface data to display different interfaces of the target application in different areas of the three-dimensional space; the second data transmission channel is used to establish an association relationship between the second virtual screen of the target device and the second texture, and the second virtual screen is used to display the response interface data in the target device.

[0124] In this exemplary embodiment, based on the user's interactive operations in the three-dimensional space, different data transmission channels can be constructed between the target device and the three-dimensional space of the virtual reality device for the display interface corresponding to each interactive operation, so that different display interfaces of the target application can be displayed simultaneously in different areas of the three-dimensional space. The construction process of each data transmission channel can refer to the construction process of the first data transmission channel and will not be repeated here. In this example, the second data transmission channel is different from the first data transmission channel, the second virtual screen is different from the first virtual screen, and the second texture is different from the first texture. Different textures can correspond to different areas of the three-dimensional space, thereby realizing the regional display of different display interfaces of the target application in the three-dimensional space, such as Figure 8 As shown, the interface data of the target application's display interface 1, display interface 2, display interface 3, and display interface 4 drawn in the corresponding virtual screens 1, virtual screen 2, virtual screen 3, and virtual screen 4 are rendered and displayed in areas A, B, C, and D in the three-dimensional space, thereby realizing split-screen display of different display interfaces in the three-dimensional space, making it convenient for users to conduct horizontal comparison and analysis of different interface contents and improving user experience.

[0125] In this exemplary embodiment, the three-dimensional space can be divided into regions based on the user's behavioral characteristics for each display interface. The user's behavioral characteristics for each display interface refer to behaviors that can reflect the user's usage habits for each display interface. For example, the behavioral characteristics may include at least one of the user's frequency of use, usage duration, dwell duration, gaze duration, time interval since last use, etc. of each display interface. Different regions of the three-dimensional space can be divided based on different user behavioral characteristics. For example, a mapping relationship can be established between a display interface with a long dwell time and the middle region of the three-dimensional space, and a mapping relationship can be established between a display interface with a short dwell time and the edge region of the three-dimensional space, so that the interface with high user attention is displayed in the middle region, making it easier for the user to capture information. The three-dimensional space corresponding to the display interface can be divided into regions based on one or more user behavioral characteristics, and this example does not limit this.

[0126] For example, if Figure 9 As shown, the interactive control method disclosed herein can be executed in a system consisting of a remote target device and a virtual reality device, and the specific steps are as follows:

[0127] S901, the virtual reality device system is started.

[0128] S902, the virtual reality device initializes the three-dimensional space of the system, that is, constructs the three-dimensional space.

[0129] S903: The virtual reality device detects a triggering operation for the target application, for example, clicking a control corresponding to the target application in the interface.

[0130] S904: The virtual reality device system calls a 3D graphics rendering tool to create a Texture and sends the corresponding TextureId to the target device.

[0131] S905: The target device creates a corresponding SurfaceTexture object based on TextureId.

[0132] S906: The target device creates a corresponding Surface object based on the SurfaceTexture object.

[0133] S907: The target device creates a corresponding virtual screen object (VirtualDisplay) based on the Surface object.

[0134] S908: The target device starts the target application.

[0135] S909: The target device runs the target application in the virtual screen object (VirtualDisplay).

[0136] S910: The target device sends the interface data including TextureId to the virtual reality device.

[0137] S911: The virtual reality device renders and displays the interface data in three-dimensional space based on TextureId.

[0138] In this example implementation, the virtual reality device can perform three-dimensional space graphics drawing by calling OpenGL ES. The virtual reality device and the target device can be devices that are physically far apart or physically close to each other. This example does not limit the actual location of each device.

[0139] In other embodiments, reference Figure 10 The present disclosure also provides an interactive display method for a virtual reality device, which can be applied to a virtual reality device. The method may include the following steps:

[0140] Step S1010: In response to detecting a startup operation for a target application, a startup interface of the target application is rendered and displayed in a target area of ​​the three-dimensional space of the virtual reality device by establishing a first data transmission channel;

[0141] Step S1020, in response to detecting a triggering operation on the first control of the startup interface, rendering and displaying the corresponding first interface data in the first area of ​​the three-dimensional space by constructing a second data transmission channel;

[0142] Step S1030, in response to detecting a triggering operation on the second control of the startup interface, rendering and displaying the corresponding second interface data in the second area of ​​the three-dimensional space by constructing a third data transmission channel;

[0143] Among them, the target application is a two-dimensional application, the target area, the first area and the second area are all different areas, the first data transmission channel is used to establish an association relationship between the startup interface and the target area, the second data transmission channel is used to establish an association relationship between the display interface corresponding to the first control and the first area, and the third data transmission channel is used to establish an association relationship between the display interface corresponding to the second control and the second area.

[0144] In this example embodiment, the target application can be a two-dimensional application installed in a virtual reality device. A three-dimensional space can be created by starting a specified application, and the user can start the target application by triggering the corresponding control of the target application. In order to realize the display of different interfaces of the target application in the three-dimensional space, a data transmission channel from the virtual screen of different interfaces to different areas of the three-dimensional space can be constructed during the triggering and displaying process of different interfaces. The construction process of the data transmission channel can be started by creating an external texture using a three-dimensional graphics drawing tool, and the corresponding virtual screen is constructed based on the external texture, so that different interfaces are displayed on the corresponding virtual screen. The specific process of constructing a virtual screen based on the external texture can be referred to the previous embodiment. In this example, the different interfaces of the two-dimensional application can be displayed in different areas in the three-dimensional space of the virtual reality device, so that the user can compare different interfaces of the same application, for example, compare the details corresponding to different products in the same shopping application, or compare the details of different characters or scenes in the same game application, thereby improving the user experience.

[0145] Based on the above embodiment, the selected area may be displayed separately or the positions of different areas may be swapped based on the user's selection or dragging operation on different display areas.

[0146] In other embodiments, based on the same inventive concept, different applications can also be displayed simultaneously in different areas of a three-dimensional space based on a data transmission channel to enable comparison of corresponding interface contents of different applications.

[0147] The following describes an embodiment of the device disclosed herein, which can be used to execute the interactive control method disclosed herein. For details not disclosed in the embodiment of the system disclosed herein, please refer to the embodiment of the interactive control method disclosed herein.

[0148] refer to Figure 11 The present disclosure further provides a virtual reality device 1110, comprising:

[0149] The sending module 1110 is configured to, in response to detecting a trigger operation for a target application, send a corresponding trigger control message to a target device, so that the target device triggers the start of the target application and sends target interface data corresponding to the target application to the virtual reality device;

[0150] a rendering module 1120 configured to, in response to receiving the target interface data, render and display the target interface data in the three-dimensional space of the virtual reality device based on the first data transmission channel; the first data transmission channel being configured to establish an association between the three-dimensional space of the virtual reality device and the display interface of the target application;

[0151] The control module 1130 is configured to send a corresponding control instruction to a target device in response to an interactive operation on a target application interface in a three-dimensional space, so as to control a target application in the target device to perform a corresponding operation response.

[0152] In some embodiments of the present disclosure, based on the aforementioned solution, the device 1100 further includes:

[0153] A message generation module is used to display at least one corresponding candidate device in response to a trigger operation on a control corresponding to a target application, where at least one candidate device is a device on which the target application is installed; in response to a selection operation on at least one candidate device, determine that the selected candidate device is a target device, and generate a trigger control message for the target application of the target device.

[0154] In some embodiments of the present disclosure, based on the aforementioned solution, the rendering module 1120 is configured to:

[0155] In response to detecting a trigger operation for a target application, creating a first texture for image drawing in a three-dimensional space;

[0156] Sending a first texture corresponding identifier to a target device, so that the target device creates a corresponding target surface texture based on the first texture corresponding identifier, creates a corresponding target surface object based on the target surface texture, creates a corresponding first virtual screen based on the target surface object, and uses the first virtual screen as a display interface of a target application to form a first data transmission channel;

[0157] Rendering the target interface data to the first texture;

[0158] In some embodiments of the present disclosure, based on the aforementioned scheme, the rendering module 1120 is also used to: construct a target model in three-dimensional space, where the target model is one of a surface model, a polygonal model, and a spherical model; and perform texture mapping on the target model according to the target interface data to render the target interface data to a first texture, and the first texture is bound to the target model.

[0159] In some embodiments of the present disclosure, based on the above solution, the sending module 1110 is further configured to:

[0160] The display parameter configuration information is sent to the target device, so that the target device configures the target display parameters of the target application, so as to bind the target application to the first virtual screen.

[0161] In some embodiments of the present disclosure, based on the above solution, the interaction control module 1130 is further configured to:

[0162] In response to an interactive operation on a target interface, a corresponding simulation trigger event is created; the simulation trigger event includes first interaction position information indicating an interaction point of the interactive operation;

[0163] The simulation trigger event is sent to the target device, so that the forwarding monitoring service in the target device forwards the monitored simulation trigger event to the first virtual screen, so that the first virtual screen responds to the simulation trigger event based on the first interaction position information.

[0164] In some embodiments of the present disclosure, based on the above solution, the rendering module 1120 is further configured to:

[0165] Receive response interface data sent by the target device and render and display the response interface data in three-dimensional space; the response interface data is generated based on the second interaction position information, and the second interaction position information is to map the interaction point to the corresponding mapping position in the first virtual screen. The second interaction position information is determined according to the boundary information of the first virtual screen and the first interaction position information.

[0166] In some embodiments of the present disclosure, based on the above solution, the interaction control module 1130 is further configured to:

[0167] Determine the first coordinates of each vertex in the target polygon corresponding to the target interface and the three-dimensional coordinates of the interaction point;

[0168] Determine the distance between the interaction point and each side of the target polygon based on the first coordinates of each vertex and the three-dimensional coordinates of the interaction point;

[0169] According to each distance and the boundary length of each side of the target polygon, the proportion of each distance in the corresponding side is determined, and the proportion is used as the first interaction position information, so that the target device determines the second interaction position information according to each proportion.

[0170] In some embodiments of the present disclosure, based on the above solution, the interaction control module 1130 is further configured to:

[0171] Before determining the first coordinates of each vertex in the target polygon, in response to the target interface not being a plane, the target interface is flattened to obtain the corresponding target polygon.

[0172] In some embodiments of the present disclosure, based on the aforementioned solution, the rendering module is further configured to:

[0173] Re-rendering the first texture using the response interface data based on the first data transmission channel to update the target interface in the three-dimensional space;

[0174] Alternatively, based on the second data transmission channel, the response interface data is used to render the second texture to display different interfaces of the target application in different areas of the three-dimensional space; the second data transmission channel is used to establish an association relationship between the second virtual screen of the target device and the second texture, and the second virtual screen is used to display the response interface data in the target device.

[0175] The specific details of each module / unit in the above-mentioned virtual reality device have been described in detail in the corresponding interactive control method, so they will not be repeated here.

[0176] refer to Figure 12 The present disclosure further provides a virtual reality device 1110, comprising:

[0177] A first rendering module 1210 is configured to, in response to detecting a startup operation for a target application, render and display a startup interface of the target application in a target area of ​​the three-dimensional space of the virtual reality device by establishing a first data transmission channel;

[0178] The second rendering module 1220 is configured to, in response to detecting a triggering operation on the first control of the startup interface, render and display the corresponding first interface data in the first area of ​​the three-dimensional space by constructing a second data transmission channel;

[0179] The third rendering module 1230 is configured to, in response to detecting a triggering operation on the second control of the startup interface, render and display the corresponding second interface data in the second area of ​​the three-dimensional space by constructing a third data transmission channel;

[0180] Among them, the target application is a two-dimensional application, the target area, the first area and the second area are all different areas, the first data transmission channel is used to establish an association relationship between the startup interface and the target area, the second data transmission channel is used to establish an association relationship between the display interface corresponding to the first control and the first area, and the third data transmission channel is used to establish an association relationship between the display interface corresponding to the second control and the second area.

[0181] The specific details of each module or unit in the above-mentioned virtual reality device have been described in detail in the corresponding interactive display method, so they will not be repeated here.

[0182] An embodiment of the present disclosure further provides an interactive control system, comprising the virtual reality device of at least one of the aforementioned embodiments.

[0183] It should be noted that although several modules or units of the device for execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0184] In this exemplary embodiment, an electronic device capable of implementing the above method is also provided.

[0185] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0186] Refer to the following Figure 13 An electronic device 1300 according to this embodiment of the present invention will be described. Figure 13 The electronic device 1300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0187] like Figure 13 As shown, electronic device 1300 is implemented as a general-purpose computing device. Components of electronic device 1300 may include, but are not limited to, the aforementioned at least one processing unit 1310, the aforementioned at least one storage unit 1320, a bus 1330 connecting various system components (including storage unit 1320 and processing unit 1310), and a display unit 1340.

[0188] The storage unit stores program codes, which can be executed by the processing unit 1310, so that the processing unit 1310 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned interactive control method and interactive display method of this specification. For example, the processing unit 1310 can execute the following steps: Figure 1 Step S110 shown in the figure: in response to detecting a trigger operation for a target application, sending a corresponding trigger control message to the target device, so that the target device triggers to start the target application and sends the target interface data corresponding to the target application to the virtual reality device; step S120: in response to receiving the target interface data, rendering and displaying the target interface data in the three-dimensional space of the virtual reality device based on the first data transmission channel to form a target interface; the first data transmission channel is used to establish an association relationship between the three-dimensional space of the virtual reality device and the display interface of the target application; step S130: in response to the interactive operation on the target interface, sending a corresponding operation control message to the target device to control the target application in the target device to perform a corresponding operation response.

[0189] The storage unit 1320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 13201 and / or a cache memory unit 13202 , and may further include a read-only memory unit (ROM) 13203 .

[0190] The storage unit 1320 may also include a program / utility 13204 having a set (at least one) of program modules 13205, such program modules 13205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0191] Bus 1330 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0192] Electronic device 1300 can also communicate with one or more external devices (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a viewer to interact with electronic device 1300, and / or any device that enables electronic device 1300 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication can occur via input / output (I / O) interface 1350. Furthermore, electronic device 1300 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1360. As shown, network adapter 1360 communicates with other modules of electronic device 1300 via bus 1330. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with electronic device 1300, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0193] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0194] In this exemplary embodiment, a computer-readable storage medium is also provided, storing a program product capable of implementing the methods described above. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0195] According to an embodiment of the present invention, a program product for implementing the above-mentioned method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, and the program can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0196] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0197] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0198] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0199] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0200] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0201] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0202] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An interactive control method for a virtual reality device, characterized in that: include: In response to detecting a trigger operation for a target application, sending a corresponding trigger control message to a target device, so that the target device triggers and starts the target application, and sends target interface data corresponding to the target application to the virtual reality device; In response to receiving the target interface data, rendering and displaying the target interface data in the three-dimensional space of the virtual reality device based on the first data transmission channel to form a target interface; The first data transmission channel is used to establish an association relationship between the three-dimensional space of the virtual reality device and the display interface of the target application; In response to the interactive operation on the target interface, a corresponding operation control message is sent to the target device to control the target application in the target device to perform a corresponding operation response.

2. The method according to claim 1, characterized in that The method further comprises: In response to a triggering operation on a control corresponding to the target application, displaying at least one corresponding candidate device, where the at least one candidate device is a device on which the target application is installed; In response to a selection operation on at least one of the candidate devices, the selected candidate device is determined to be the target device, and the trigger control message for the target application of the target device is generated.

3. The method according to claim 1, characterized in that The method further comprises: In response to detecting a trigger operation for a target application, creating a first texture for image drawing in a three-dimensional space; sending the first texture corresponding identifier to a target device, so that the target device creates a corresponding target surface texture based on the first texture corresponding identifier, creates a corresponding target surface object based on the target surface texture, creates a corresponding first virtual screen based on the target surface object, and uses the first virtual screen as a display interface of the target application to form the first data transmission channel; Rendering and displaying the target interface data in the three-dimensional space of the virtual reality device based on the first data transmission channel includes: Rendering the target interface data to the first texture.

4. The method according to claim 3, characterized in that Rendering the target interface data to the first texture includes: Constructing a target model in a three-dimensional space, wherein the target model is one of a surface model, a polygonal model, and a spherical model; According to the target interface data, texture mapping is performed on the target model to render the target interface data to a first texture, and the first texture is bound to the target model.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Display parameter configuration information is sent to the target device, so that the target device configures target display parameters of the target application, so as to bind the target application to the first virtual screen.

6. The method according to claim 4, characterized in that: Sending a corresponding operation control instruction to the target device to control a target application in the target device to perform a corresponding operation response, including: In response to the interactive operation on the target interface, a corresponding simulation trigger event is created; the simulation trigger event includes first interaction position information for indicating an interaction point of the interactive operation; The simulation trigger event is sent to the target device, so that the forwarding monitoring service in the target device forwards the monitored simulation trigger event to the first virtual screen, so that the first virtual screen responds to the simulation trigger event based on the first interaction position information.

7. The method according to claim 4, characterized in that: The method further comprises: Receive response interface data sent by the target device, and render and display the response interface data in three-dimensional space; the response interface data is generated based on second interaction position information, the second interaction position information is to map the interaction point to the corresponding mapping position in the first virtual screen, and the second interaction position information is determined according to the boundary information of the first virtual screen and the first interaction position information.

8. The method according to claim 7, characterized in that: The method further comprises: Determine the first coordinates of each vertex in the target polygon corresponding to the target interface and the three-dimensional coordinates of the interaction point; Determine the distance between the interaction point and each side of the target polygon according to the first coordinates of each vertex and the three-dimensional coordinates of the interaction point; According to each of the distances and the boundary length of each side of the target polygon, the proportion of each distance in the corresponding side is determined, and the proportion is used as the first interaction position information, so that the target device determines the second interaction position information according to each proportion.

9. The method according to claim 8, characterized in that Before determining the first coordinates of each vertex in the target polygon, the method further includes: In response to the target interface not being a plane, the target interface is flattened to obtain a corresponding target polygon.

10. The method according to claim 7, characterized in that: The method further comprises: Re-rendering the first texture using the response interface data based on the first data transmission channel to update the target interface in the three-dimensional space; Alternatively, based on a second data transmission channel, the response interface data is used to render a second texture to display different interfaces of the target application in different areas of the three-dimensional space; the second data transmission channel is used to establish an association relationship between the second virtual screen of the target device and the second texture, and the second virtual screen is used to display the response interface data in the target device.

11. An interactive display method for a virtual reality device, characterized in that: include: In response to detecting a startup operation for a target application, rendering and displaying a startup interface of the target application in a target area of ​​the three-dimensional space of the virtual reality device by establishing a first data transmission channel; In response to detecting a triggering operation on a first control of the startup interface, rendering and displaying corresponding first interface data in a first area of ​​the three-dimensional space by constructing a second data transmission channel; In response to detecting a triggering operation on a second control of the startup interface, rendering and displaying corresponding second interface data in a second area of ​​the three-dimensional space by constructing a third data transmission channel; In which, the target application is a two-dimensional application, the target area, the first area and the second area are all different areas, the first data transmission channel is used to establish an association relationship between the startup interface and the target area, the second data transmission channel is used to establish an association relationship between the display interface corresponding to the first control and the first area, and the third data transmission channel is used to establish an association relationship between the display interface corresponding to the second control and the second area.

12. A virtual reality device, characterized in that: include: a sending module, configured to, in response to detecting a trigger operation for a target application, send a corresponding trigger control message to a target device, so that the target device triggers and starts the target application, and sends target interface data corresponding to the target application to the virtual reality device; A rendering module, configured to render and display the target interface data in a three-dimensional space of the virtual reality device based on a first data transmission channel in response to receiving the target interface data; The first data transmission channel is used to establish an association relationship between the three-dimensional space of the virtual reality device and the display interface of the target application; The interactive control module is used to respond to the interactive operation on the target application interface in the three-dimensional space and send a corresponding control instruction to the target device to control the target application in the target device to perform a corresponding operation response.

13. A virtual reality device, characterized in that: include: a first rendering module, configured to, in response to detecting a startup operation for a target application, render and display a startup interface of the target application in a target area of ​​the three-dimensional space of the virtual reality device by establishing a first data transmission channel; A second rendering module is configured to, in response to detecting a triggering operation on the first control of the startup interface, render and display the corresponding first interface data in a first area of ​​the three-dimensional space by constructing a second data transmission channel; a third rendering module, configured to, in response to detecting a triggering operation on the second control of the startup interface, render and display the corresponding second interface data in the second area of ​​the three-dimensional space by constructing a third data transmission channel; In which, the target application is a two-dimensional application, the target area, the first area and the second area are all different areas, the first data transmission channel is used to establish an association relationship between the startup interface and the target area, the second data transmission channel is used to establish an association relationship between the display interface corresponding to the first control and the first area, and the third data transmission channel is used to establish an association relationship between the display interface corresponding to the second control and the second area.

14. An interactive control system, characterized in that: include: The virtual reality device according to claim 12 or 13.

15. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, causes the one or more processors to implement the method according to any one of claims 1 to 11.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images