Interaction control method and apparatus, and electronic device and medium

Abstract

Disclosed in the present invention are an interaction control method and apparatus, and an electronic device and a medium. The method comprises: receiving a touch-control operation performed by a user; when there is a collision point between a virtual identifier and a first application in an application list, in response to the touch-control operation, creating a first virtual screen and a first window, running the first application on the first virtual screen, and rendering screen data of the first virtual screen to the first window; acquiring degree-of-freedom information of a head-mounted display device; and adjusting the first window on the basis of the degree-of-freedom information, and encode the screen data in the adjusted first window, and then sending same to the head-mounted display device for decoding and display.

Description

Interactive control method, device, electronic device and medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 16, 2023, with application number 202311032642.5 and invention name “Interactive control method, device, electronic device and medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the technical field of wearable devices, and more specifically, to an interactive control method, an interactive control device, an electronic device, and a computer-readable storage medium. Background Art

[0003] Split-type displays are becoming a key trend in AR development. To reduce power consumption, AR glasses are currently connected to mobile devices, with content on the mobile device being transferred to the AR glasses for display. However, in related technologies, there is a problem with how to transfer different content on a mobile device to the AR glasses for display.

[0004] Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide an interactive control method, device, electronic device and medium.

[0006] According to a first aspect of an embodiment of the present invention, there is provided an interactive control method, the method comprising:

[0007] Receive user touch operations;

[0008] When there is a collision point between the virtual identifier and the first application in the application list, in response to the touch operation, creating a first virtual screen and a first window, running the first application on the first virtual screen, and rendering screen data of the first virtual screen to the first window;

[0009] Obtaining the degree of freedom information of the head-mounted display device;

[0010] The first window is adjusted according to the degree of freedom information, and the screen data in the adjusted first window is encoded and sent to the head-mounted display device for decoding and display.

[0011] Optionally, after adjusting the first window according to the degree of freedom information and encoding the screen data in the adjusted first window and sending it to the head-mounted display device for decoding and display, the method further includes:

[0012] When a touch event is received, obtaining position information of a collision point between the virtual marker and the first window;

[0013] determining relative position information of the collision point within the first virtual screen according to the position information of the collision point;

[0014] The touch event is distributed to the first virtual screen for response according to the relative position information.

[0015] Optionally, adjusting the first window according to the degree of freedom information, and encoding and sending the screen data in the adjusted first window to the head-mounted display device for decoding and display, includes:

[0016] The first window is adjusted according to the degree of freedom information, and an edge area of ​​the screen data in the adjusted first window is compressed and sent to the head-mounted display device for decoding and display.

[0017] Optionally, adjusting the first window according to the degree of freedom information, and encoding and sending the screen data in the adjusted first window to the head-mounted display device for decoding and display, includes:

[0018] adjusting the first window and the second window according to the degree of freedom information;

[0019] Combining and encoding the adjusted screen data in the first window and the adjusted screen data in the second window, and sending the combined data to the head-mounted display device for decoding and display;

[0020] The second virtual screen runs the second application, and the second window renders screen data of the second virtual screen.

[0021] Optionally, the method further includes:

[0022] Get the posture information of the terminal device;

[0023] According to the posture information, the virtual marker is controlled to rotate so that a collision point exists between the virtual marker and a target application in the application list, or a collision point exists between the virtual marker and the first window;

[0024] The target application is the first application or the second application.

[0025] According to a second aspect of an embodiment of the present invention, there is provided an interactive control device, the device comprising:

[0026] A receiving module, used to receive a user's touch operation;

[0027] a creation module configured to, when a collision point exists between the virtual identifier and a first application in the application list, create a first virtual screen and a first window in response to the touch operation, run the first application on the first virtual screen, and render screen data of the first virtual screen to the first window;

[0028] An acquisition module, used to obtain degree of freedom information of a head-mounted display device;

[0029] A sending module is used to adjust the first window according to the degree of freedom information, and encode the screen data in the adjusted first window and send it to the head-mounted display device for decoding and display.

[0030] Optionally, the device further includes a determination module,

[0031] The acquisition module is further configured to acquire, upon receiving a touch event, position information of a collision point between the virtual marker and the first window;

[0032] a determination module, configured to determine relative position information of the collision point within the first virtual screen according to the position information of the collision point;

[0033] The sending module is configured to distribute the touch event to the first virtual screen for response according to the relative position information.

[0034] Optionally, the sending module is specifically configured to:

[0035] The first window is adjusted according to the degree of freedom information, and an edge area of ​​the screen data in the adjusted first window is compressed and sent to the head-mounted display device for decoding and display.

[0036] Optionally, the sending module is specifically configured to:

[0037] adjusting the first window and the second window according to the degree of freedom information;

[0038] Combining and encoding the adjusted screen data in the first window and the adjusted screen data in the second window, and sending the combined data to the head-mounted display device for decoding and display;

[0039] The second virtual screen runs the second application, and the second window renders screen data of the second virtual screen.

[0040] Optionally, the device further includes a control module,

[0041] The acquisition module is further used to obtain the posture information of the terminal device;

[0042] a control module, configured to control the virtual identifier to rotate according to the posture information so that a collision point exists between the virtual identifier and a target application in the application list, or a collision point exists between the virtual identifier and the first window;

[0043] The target application is the first application or the second application.

[0044] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising:

[0045] Memory for storing executable computer instructions;

[0046] The processor is configured to execute the interactive control method according to the first aspect above under the control of the executable computer instructions.

[0047] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the interactive control method described in the first aspect is executed.

[0048] One advantageous effect of an embodiment of the present invention is that, upon receiving a user's touch operation, if there is a collision point between the virtual identifier and the first application in the application list, the terminal device can respond to the touch operation by creating a first virtual screen and a first window, running the first application on the first virtual screen and rendering the screen data of the first virtual screen to the first window, obtaining the degrees of freedom information of the head-mounted display device, adjusting the first window based on the degrees of freedom information, and then encoding the screen data in the adjusted first window and sending it to the head-mounted display device for decoding and display. In other words, the user can transmit different application scenarios rendered by the terminal device to the head-mounted display device for display according to their needs.

[0049] Other features and advantages of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0051] FIG1 is a schematic diagram of the hardware configuration of an interactive control system according to an embodiment of the present invention;

[0052] FIG2 is a schematic flow chart of an interactive control method according to an embodiment of the present invention;

[0053] 3 is a schematic diagram of an interface display of a terminal device according to an embodiment of the present invention;

[0054] 4 is a schematic diagram of an interface display of a terminal device according to another embodiment of the present invention;

[0055] FIG5 is a functional block diagram of an interactive control device according to an embodiment of the present invention;

[0056] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0059] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme in which A and B are satisfied at the same time. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0060] <Hardware Configuration>

[0061] Figure 1 is a schematic diagram of the hardware configuration of an interactive control system that can be used to implement an interactive control method according to one embodiment. Figure 1 shows a head-mounted display device 100, a terminal device 200, and a network 300. The head-mounted display device 100 can be connected to the network 300 and can also be connected to the terminal device 200 via a communication method such as Bluetooth. In one embodiment, the head-mounted display device 100 is connected to the terminal device 200 only via a communication method such as Bluetooth. Multiple servers 301 and 302 can be provided in the network 300. The network 300 can be a wireless communication network or a wired communication network. The network 300 can be a local area network or a wide area network. The network 300 can be used for short-range or long-range communication.

[0062] In one embodiment, as shown in FIG1 , a head-mounted display device 100 may include a processor 101 and a memory 102. The head-mounted display device 100 may also include a communication device 103, a display device 104, a user interface 105, a camera 106, an audio / video interface 107, and a sensor 108. Furthermore, the head-mounted display device 100 may also include a power management chip 109 and a battery 110.

[0063] The processor 101 may be any of a variety of processors. The memory 102 may store the underlying software, system software, application software, data, and other components required for the operation of the head-mounted display device 100. The memory 102 may include various memory types, such as ROM, RAM, and Flash. The communication device 103 may include, for example, WiFi, Bluetooth, 3G, 4G, or 5G communication devices. Through the communication device 103, the head-mounted display device 100 can be deployed in a network. The display device 104 may be, for example, an LCD or OLED display. In one example, the display device 104 may be a touch screen. Users can perform input operations through the display device 104. Furthermore, users can also perform fingerprint recognition and other functions through the touch screen. The user interface 105 may include, for example, a USB port, a Lightning port, a keyboard, and the like. The camera device 106 may include a single camera or multiple cameras. The audio / video interface 107 may include, for example, a speaker port, a microphone port, and a video transmission port such as HDMI. The sensor 108 may include, for example, a gyroscope, an accelerometer, a temperature sensor, a humidity sensor, a pressure sensor, and the like. For example, these sensors can be used to determine the posture information of the head-mounted display device. The power management chip 109 can be used to manage the power input to the head mounted display device 100 and can also manage the battery 110 to ensure greater utilization efficiency. The battery 110 is, for example, a lithium-ion battery.

[0064] The head-mounted display device 100 may be AR (Augmented Reality) glasses and MR (Mixed Reality) glasses, etc. The AR glasses may be split AR glasses, and the MR glasses may be split MR glasses, which are not limited in this embodiment of the present invention. The various components shown in Figure 1 are merely schematic. The head-mounted display device 100 may include one or more of the components shown in Figure 1, but does not necessarily include all of the components in Figure 1. The head-mounted display device 100 shown in Figure 1 is merely illustrative and is in no way intended to limit the embodiments herein, their applications, or uses.

[0065] In one embodiment, as shown in FIG1 , a terminal device 200 may include a processor 201 and a memory 202. The terminal device 200 may also include a communication device 203, a display device 204, a user interface 205, a camera 206, an audio / video interface 207, a sensor 208, and the like. Furthermore, the terminal device 200 may also include a power management chip 209 and a battery 210, and the like.

[0066] The terminal device 200 may be a mobile phone, a portable computer, a tablet computer, a PDA, a wearable device, etc., and this is not limited in the embodiments of the present invention. For example, the terminal device may be an Android device. For example, the terminal device may be an Android device with a display device 204, or the terminal device may be an Android device without a display device 204. In other words, the various components shown in FIG1 are merely illustrative. The terminal device 200 may include one or more of the components shown in FIG1 , but does not necessarily include all of the components shown in FIG1 . The terminal device 200 shown in FIG1 is merely illustrative and is in no way intended to limit the embodiments herein, their applications, or uses.

[0067] In this embodiment, the memory 202 of the terminal device 200 is used to store program instructions. These program instructions are used to control the processor 201 to perform the interactive control method. A skilled person can design these instructions based on the disclosed solution. How instructions control the processor's operation is well known in the art and will not be described in detail here.

[0068] It should be understood that although FIG1 shows only one head-mounted display device 100 and terminal device 200, it does not mean to limit their respective numbers. The interactive control system may include multiple head-mounted display devices 100 and multiple terminal devices 200.

[0069] In the above description, a person skilled in the art can design instructions based on the solution provided by the present invention. How instructions control the operation of a processor is well known in the art and will not be described in detail here.

[0070] <Method Example>

[0071] FIG2 shows an interactive control method according to an embodiment of the present invention. The interactive control method can be implemented by the terminal device shown in FIG1 , which can be, for example, an Android device without a display device 204 . As shown in FIG2 , the interactive control method according to this embodiment can include the following steps S2100 to S2400 :

[0072] Step S2100: receiving a touch operation from the user.

[0073] The touch operation may be a touch operation performed by the user on a touch pad of the terminal device to start the first application.

[0074] In this embodiment, the desktop launcher of the terminal device is called Launcher, which is automatically started when the terminal device is turned on. The physical screen of the terminal device runs a control application, which usually runs on the front end of the terminal device to ensure the normal use of the touchpad of the terminal device.

[0075] After executing S2100 to receive the user's touch operation, enter:

[0076] Step S2200: When there is a collision point between the virtual identifier and the first application in the application list, in response to the touch operation, a first virtual screen and a first window are created, the first application is run on the first virtual screen, and the screen data of the first virtual screen is rendered to the first window.

[0077] The virtual identifier can represent the current posture of the terminal device, and the user can control the terminal device based on the virtual identifier. For example, the virtual identifier can be a virtual ray. The virtual ray can be a straight line or a curve, which is not limited in this embodiment.

[0078] In one embodiment, when a terminal device establishes a streaming connection with a head-mounted display, it renders and displays an application list through the Launcher. Referring to Figure 3, the application list displays icons for the control application, application 1, and application 2. Simultaneously, based on the established streaming connection with the head-mounted display, the terminal device encodes and sends the rendered application list to the head-mounted display for decoding and display.

[0079] Among them, the streaming connection established between the terminal device and the head-mounted display device can be a wired streaming connection or a wireless streaming connection, which is not limited in this embodiment.

[0080] Specifically, when the terminal device detects a collision point between the virtual identifier and the icon of the first application in the application list displayed by the head-mounted display device, it can respond to the first touch operation by creating a first virtual screen and a first window in the Launcher, running the first application on the first virtual screen, and obtaining left screen data and right screen data based on the screen data of the first virtual screen and rendering them to the first window. For example, referring to Figure 4, the first virtual screen runs application 1, and the first window renders the screen data of the first virtual screen.

[0081] Step S2300: Obtain degree of freedom information of the head mounted display device.

[0082] Among them, the degree of freedom information of the head-mounted display device refers to the six degrees of freedom information of the head-mounted display device (hereinafter referred to as 6DoF information), including the degree of freedom of movement and the degree of freedom of rotation.

[0083] In one embodiment, after the camera device of the head-mounted display device captures the first image, the head-mounted display device uses a simultaneous localization and mapping (SLAM) algorithm to perform calculations based on spatial structure information in the first image to obtain the head-mounted display device's movement degrees of freedom (i.e., three-dimensional coordinates); based on the head-mounted display device's posture information (e.g., quaternion information) captured by the head-mounted display device's inertial measurement unit, the head-mounted display device's rotational degrees of freedom (i.e., yaw angle, pitch angle, and roll angle) can be obtained; and the head-mounted display device's 6DoF information can be determined using the movement degrees of freedom and the rotational degrees of freedom.

[0084] Specifically, after the head-mounted display device obtains the degree of freedom information of the head-mounted display device, it sends the degree of freedom information of the head-mounted display device to the terminal device.

[0085] Step S2400: Adjust the first window according to the degree of freedom information, and encode the screen data in the adjusted first window and send it to the head-mounted display device for decoding and display.

[0086] Specifically, the terminal device adjusts the first window according to the degree of freedom information of the head-mounted display device, and sends the screen data rendered by the first window to the head-mounted display device for decoding and display. That is, the head-mounted display device only serves as a display device of the terminal device to display screen data.

[0087] In one embodiment, this step S2400 adjusts the first window according to the degree of freedom information, and encodes the screen data in the adjusted first window and sends it to the head-mounted display device for decoding and display, which may further include: adjusting the first window according to the degree of freedom information, and compressing the edge area of ​​the screen data in the adjusted first window and sending it to the head-mounted display device for decoding and display.

[0088] In this embodiment, when the amount of screen data in the adjusted first window is greater than the preset data amount, since the head-mounted display device itself has lens distortion, the terminal device will compress the edge area of ​​the screen data in the first window and send it to the head-mounted display device for decoding and display.

[0089] In one embodiment, this step S2400 adjusts the first window according to the degree of freedom information, and encodes the screen data in the adjusted first window and sends it to the head-mounted display device for decoding and display, which may further include: adjusting the first window and the second window according to the degree of freedom information; and encoding the screen data in the adjusted first window and the screen data in the adjusted second window together and sending them to the head-mounted display device for decoding and display.

[0090] The second virtual screen runs a second application, and the second window renders screen data of the second virtual screen.

[0091] It should be noted that the terminal device can also receive a touch operation from the user, wherein the touch operation can be a touch operation performed by the user on the touchpad of the terminal device to start the second application. When the terminal device detects that there is a collision point between the virtual identifier and the icon of the second application in the application list, it can respond to the touch operation, create a second virtual screen and a second window, run the second application on the second virtual screen, and obtain left screen data and right screen data based on the screen data of the second virtual screen and render them to the second window. In other words, the control application in the terminal device will run on the physical screen, and each other application on the terminal device will run on the corresponding virtual screen. This embodiment does not specifically limit the number of applications.

[0092] Exemplarily, referring to FIG. 4 , the second virtual screen runs application 2 , and the second window renders screen data of the second virtual screen.

[0093] In this embodiment, the terminal device adjusts the first window and the second window respectively according to the degree of freedom information of the head-mounted display device, and merges and encodes the screen data rendered by the adjusted first window and the screen data rendered by the adjusted second window and sends them to the head-mounted display device for decoding and display.

[0094] According to this embodiment, upon receiving a user's touch operation, if there is a collision point between the virtual identifier and the first application in the application list, the terminal device can respond to the touch operation by creating a first virtual screen and a first window, running the first application on the first virtual screen and rendering the screen data of the first virtual screen into the first window. Furthermore, the terminal device obtains the degrees of freedom information of the head-mounted display device, adjusts the first window based on the degrees of freedom information, and then encodes the screen data in the adjusted first window and transmits it to the head-mounted display device for decoding and display. In other words, the user can transmit different application scenarios rendered by the terminal device to the head-mounted display device for display according to their needs.

[0095] In one embodiment, after executing step S2300 above to adjust the first window according to the degree of freedom information, and encoding the screen data in the adjusted first window and sending it to the head-mounted display device for decoding and display, the interactive control method of this embodiment of the present invention further includes the following steps S3100 to S3300:

[0096] Step S3100: When a touch event is received, obtain position information of a collision point between the virtual marker and the first window.

[0097] The touch event may be a press event, a slide event, or the like.

[0098] In this embodiment, when the terminal device receives a touch event performed by the user on the touchpad, if it detects that there is a collision point between the virtual marker and the first window displayed by the head-mounted display device, the position information of the collision point between the virtual marker and the first window can be obtained.

[0099] Step S3200: Determine relative position information of the collision point within the first virtual screen according to the position information of the collision point.

[0100] Step S3300: Distribute the touch event to the first virtual screen for response according to the relative position information.

[0101] According to this embodiment, the terminal device can distribute the touch event implemented by the user to the corresponding virtual screen for response.

[0102] In one embodiment, the interactive control method of the embodiment of the present invention further includes the following steps S4100 to S4200:

[0103] Step S4100, obtaining the posture information of the terminal device.

[0104] In this embodiment, the posture information (eg, quaternion information) of the terminal device is collected according to the inertial measurement unit of the terminal device.

[0105] Step S4200: Control the virtual marker to rotate according to the posture information so that a collision point exists between the virtual marker and a target application in the application list, or between the virtual marker and the first window.

[0106] The target application is the first application or the second application.

[0107] In this embodiment, the terminal device obtains the rotational degrees of freedom (i.e., yaw, pitch, and roll) of the head-mounted display device based on the posture information, and the rotational degrees of freedom can be used as the 3DoF information of the terminal device. The virtual identifier is then controlled to rotate based on the 3DoF information so that a collision point exists between the virtual identifier and the icon of the first application in the application list, or a collision point exists between the virtual identifier and the icon of the second application in the application list, or a collision point exists between the virtual identifier and the first window.

[0108] <Device Example>

[0109] FIG5 is a schematic diagram of the principles of an interactive control device according to an embodiment. As shown in FIG5 , the interactive control device 500 includes a receiving module 510 , a creating module 520 , an acquiring module 530 , and a sending module 540 .

[0110] Receiving module 510, for receiving a user's touch operation;

[0111] a creation module 520 configured to, when a collision point exists between the virtual identifier and a first application in the application list, create a first virtual screen and a first window in response to the touch operation, run the first application on the first virtual screen, and render screen data of the first virtual screen to the first window;

[0112] An acquisition module 530 is configured to acquire degree of freedom information of a head mounted display device;

[0113] The sending module 540 is configured to adjust the first window according to the degree of freedom information, and encode and send the screen data in the adjusted first window to the head mounted display device for decoding and display.

[0114] In one embodiment, the apparatus further includes a determination module (not shown in the figure).

[0115] The acquisition module 530 is further configured to acquire, upon receiving a touch event, location information of a collision point between the virtual marker and the first window;

[0116] a determination module, configured to determine relative position information of the collision point within the first virtual screen according to the position information of the collision point;

[0117] The sending module 540 is configured to distribute the touch event to the first virtual screen for response according to the relative position information.

[0118] In one embodiment, the sending module 540 is specifically configured to adjust the first window according to the degree of freedom information, and compress an edge area of ​​the screen data in the adjusted first window and send the compressed data to the head mounted display device for decoding and display.

[0119] In one embodiment, the sending module 540 is specifically used to: adjust the first window and the second window according to the degree of freedom information; merge and encode the screen data in the adjusted first window and the screen data in the adjusted second window and send them to the head-mounted display device for decoding and display; wherein, the second virtual screen runs the second application, and the second window renders the screen data of the second virtual screen.

[0120] In one embodiment, the device further includes a control module (not shown in the figure).

[0121] The acquisition module 530 is further used to obtain the posture information of the terminal device;

[0122] a control module, configured to control the virtual identifier to rotate according to the posture information so that a collision point exists between the virtual identifier and a target application in the application list, or a collision point exists between the virtual identifier and the first window;

[0123] The target application is the first application or the second application.

[0124] According to an embodiment of the present invention, upon receiving a user's touch operation, if there is a collision point between the virtual identifier and the first application in the application list, the terminal device can respond to the touch operation by creating a first virtual screen and a first window, running the first application on the first virtual screen and rendering the screen data of the first virtual screen into the first window, obtaining the degrees of freedom information of the head-mounted display device, adjusting the first window based on the degrees of freedom information, and then encoding the adjusted screen data in the first window and sending it to the head-mounted display device for decoding and display. In other words, the user can transmit different application scenarios rendered by the terminal device to the head-mounted display device for display according to their needs.

[0125] <Equipment Example>

[0126] FIG6 is a schematic diagram of the hardware structure of an electronic device according to an embodiment. As shown in FIG6 , the electronic device 600 includes a processor 610 and a memory 620 .

[0127] The memory 620 may be used to store executable computer instructions.

[0128] The processor 610 can be used to execute the interactive control method according to the method embodiment of the present invention under the control of the executable computer instructions.

[0129] The electronic device 600 may be the head mounted display device 100 as shown in FIG. 1 , or the terminal device 200 as shown in FIG. 1 , or a device with other hardware structures, which is not limited here.

[0130] In another embodiment, the electronic device 600 may include the above interactive control device 500 .

[0131] In one embodiment, each module of the above interactive control device 600 can be implemented by the processor 610 running computer instructions stored in the memory 620 .

[0132] <Computer-readable storage medium>

[0133] An embodiment of the present invention further provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the interactive control method provided by the embodiment of the present invention is executed.

[0134] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0135] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0136] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0137] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0138] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0140] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0141] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent. It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0142] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0143] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as mentioned above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0144] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. An interactive control method, characterized in that: The method comprises: Receive user's touch operation; When there is a collision point between the virtual identifier and the first application in the application list, in response to the touch operation, creating a first virtual screen and a first window, running the first application on the first virtual screen, and rendering screen data of the first virtual screen to the first window; Obtaining the degree of freedom information of the head mounted display device; The first window is adjusted according to the degree of freedom information, and the screen data in the adjusted first window is encoded and sent to the head mounted display device for decoding and display.

2. The method according to claim 1, characterized in that After adjusting the first window according to the degree of freedom information, and encoding the screen data in the adjusted first window and sending it to the head mounted display device for decoding and display, the method further includes: When a touch event is received, obtaining position information of a collision point between the virtual marker and the first window; Determining relative position information of the collision point within the first virtual screen according to the position information of the collision point; According to the relative position information, the touch event is distributed to the first virtual screen for response.

3. The method according to claim 1, characterized in that The step of adjusting the first window according to the degree of freedom information, and encoding the screen data in the adjusted first window and sending it to the head mounted display device for decoding and display, comprises: The first window is adjusted according to the degree of freedom information, and an edge area of ​​the screen data in the adjusted first window is compressed and sent to the head mounted display device for decoding and display.

4. The method according to claim 2, characterized in that: The step of adjusting the first window according to the degree of freedom information, and encoding the screen data in the adjusted first window and sending it to the head mounted display device for decoding and display, comprises: adjusting the first window and the second window according to the degree of freedom information; The adjusted screen data in the first window and the adjusted screen data in the second window are combined and encoded and sent to the head mounted display device for decoding and display; The second virtual screen runs the second application, and the second window renders screen data of the second virtual screen.

5. The method according to claim 4, characterized in that The method further comprises: Get the posture information of the terminal device; According to the posture information, controlling the virtual identifier to rotate so that a collision point exists between the virtual identifier and a target application in the application list, or a collision point exists between the virtual identifier and the first window; The target application is the first application or the second application.

6. An interactive control device, characterized in that: The device comprises: A receiving module, used for receiving a user's touch operation; a creation module, configured to create a first virtual screen and a first window in response to the touch operation when there is a collision point between the virtual identifier and the first application in the application list, run the first application on the first virtual screen, and render screen data of the first virtual screen to the first window; An acquisition module, used to obtain degree of freedom information of a head mounted display device; A sending module is used to adjust the first window according to the degree of freedom information, and encode the screen data in the adjusted first window and send it to the head mounted display device for decoding and display.

7. The device according to claim 6, characterized in that The device also includes a determination module, The acquisition module is further configured to acquire, when receiving a touch event, position information of a collision point between the virtual marker and the first window; A determination module, configured to determine relative position information of the collision point within the first virtual screen according to the position information of the collision point; The sending module is used to distribute the touch event to the first virtual screen for response according to the relative position information.

8. The device according to claim 6, characterized in that The sending module is specifically used for: The first window is adjusted according to the degree of freedom information, and an edge area of ​​the screen data in the adjusted first window is compressed and sent to the head mounted display device for decoding and display.

9. The device according to claim 7, characterized in that The sending module is specifically used for: adjusting the first window and the second window according to the degree of freedom information; The adjusted screen data in the first window and the adjusted screen data in the second window are combined and encoded and sent to the head mounted display device for decoding and display; The second virtual screen runs the second application, and the second window renders screen data of the second virtual screen.

10. The device according to claim 9, characterized in that The device also includes a control module, The acquisition module is also used to acquire the posture information of the terminal device; a control module, configured to control the virtual identifier to rotate according to the posture information, so that a collision point exists between the virtual identifier and a target application in the application list, or a collision point exists between the virtual identifier and the first window; The target application is the first application or the second application.

11. An electronic device, characterized in that: The electronic device comprises: Memory for storing executable computer instructions; A processor is used to execute the interactive control method according to any one of claims 1-5 under the control of the executable computer instructions.

12. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are executed by a processor to execute the interactive control method according to any one of claims 1 to 5.

Images