Interaction control method and apparatus, head-mounted display device, and medium

Abstract

The present disclosure provides an interaction control method and device, a head-mounted display device and a medium. The method comprises: in the case of receiving a touch event, obtaining a touch position of the touch event; if the touch position is located in one of a plurality of virtual screens of a head-mounted display device, obtaining a relative position of the touch position in the one of the virtual screens, wherein the head-mounted display device comprises a plurality of virtual screens, and the plurality of virtual screens are located at different positions of a display area of the head-mounted display device; and according to the relative position and attribute information of the one of the virtual screens, re-encapsulating and distributing the touch event to the one of the virtual screens for response.

Description

Technical Field

[0001] This disclosure relates to the field of wearable device technology, and more specifically, to an interactive control method, an interactive control device, a head-mounted display device, and a computer-readable storage medium. Background Technology

[0002] For the Android system, only one application can be displayed in the foreground at a time on a given screen. When a second application is opened, the first application will be pushed to the background.

[0003] In related technologies, the Android system can support multiple display screens and virtual screen functionality. This means that the Android system can support multiple display screens existing simultaneously, and the virtual screen can be displayed on top of the default Android screen. Therefore, the ability of head-mounted display devices, such as AR glasses, to run multiple applications simultaneously can take advantage of the virtual screen feature. When multiple applications are opened at the same time, each application can create a corresponding virtual screen, and each application is displayed separately on its corresponding virtual screen and displayed at the forefront of the AR glasses screen. This can meet the needs of running multiple applications simultaneously.

[0004] Although the above solution can enable multiple applications to run simultaneously, in the Android system's touch event distribution and response process, a touch event can only be distributed to one display screen, which is the default Android screen (DefaultDisplay) by default. Although when a user touches the virtual screen display area where an application is located, it visually appears to be within the range of this virtual screen, it actually still belongs to the default screen (DefaultDisplay). This results in the final touch event not being distributed to the corresponding virtual screen. Summary of the Invention

[0005] One objective of this disclosure is to provide a new technical solution for interactive control.

[0006] According to a first aspect of the present disclosure, an interactive control method is provided, the method comprising:

[0007] Upon receiving a touch event, the touch location of the touch event is obtained;

[0008] If the touch position is located within one of the multiple virtual screens of the head-mounted display device, then the relative position of the touch position within that virtual screen is obtained, wherein the head-mounted display device includes multiple virtual screens, and the multiple virtual screens are located at different positions in the display area of ​​the head-mounted display device;

[0009] Based on the relative position and the attribute information of one of the virtual screens, the touch event is repackaged and distributed to one of the virtual screens for response.

[0010] Optionally, the method further includes:

[0011] Obtain the set mapping data; wherein, the mapping data includes attribute information of different virtual screens and their corresponding position information;

[0012] Traverse the position information corresponding to each attribute information in the mapping data;

[0013] Based on the touch position and the position information corresponding to the traversed attribute information, determine whether the touch position is located within the virtual screen indicated by the traversed attribute information.

[0014] Optionally, determining whether the touch position is located within the virtual screen indicated by the traversed attribute information based on the touch position and the position information corresponding to the traversed attribute information includes:

[0015] If the horizontal coordinate position of the touch position is located between the upper left and lower right horizontal coordinate positions of the traversed location information, and the vertical coordinate position of the touch position is located between the upper left and lower right vertical coordinate positions of the traversed location information, then the touch position is determined to be within the virtual screen indicated by the traversed attribute information.

[0016] Optionally, the relative position of the touch position within one of the virtual screens includes: the horizontal relative position of the touch position within one of the virtual screens, and the vertical relative position of the touch position within one of the virtual screens.

[0017] The step of obtaining the relative position of the touch location within one of the virtual screens includes:

[0018] Based on the horizontal coordinates of the touch location and the horizontal coordinates of the top left corner of one of the virtual screens, the horizontal relative position of the touch location within one of the virtual screens is obtained;

[0019] Based on the vertical coordinates of the touch position and the vertical coordinates of the top left corner of one of the virtual screens, the vertical relative position of the touch position within one of the virtual screens is obtained.

[0020] Optionally, the method further includes:

[0021] The plurality of virtual screens are created in the display area of ​​the head-mounted display device;

[0022] Record the attribute and position information of each virtual screen to obtain mapping data;

[0023] Store the mapping data;

[0024] The location information of the virtual screen includes: the coordinates of the upper left corner of the virtual screen and the coordinates of the lower right corner of the virtual screen;

[0025] The coordinates of the upper left corner of the virtual screen include: the horizontal coordinates of the upper left corner of the virtual screen and the vertical coordinates of the upper left corner of the virtual screen.

[0026] The coordinates of the lower right corner of the virtual screen include: the horizontal coordinates of the lower right corner of the virtual screen and the vertical coordinates of the lower right corner of the virtual screen.

[0027] According to a second aspect of the present disclosure, a display control device is provided, the device comprising:

[0028] The receiving module is used to receive touch events;

[0029] The first acquisition module is used to acquire the touch position of the touch event;

[0030] The second acquisition module is used to acquire the relative position of the touch position within one of the multiple virtual screens of the head-mounted display device if the touch position is located within one of the multiple virtual screens of the head-mounted display device. The head-mounted display device includes multiple virtual screens, and the multiple virtual screens are located at different positions in the display area of ​​the head-mounted display device.

[0031] An encapsulation module is used to re-encapsulate and distribute the touch event to one of the virtual screens for response based on the relative position and the attribute information of one of the virtual screens.

[0032] According to a third aspect of the present disclosure, a head-mounted display device is provided, the head-mounted display device comprising: a memory for storing executable computer instructions; and a processor for executing the interactive control method according to the first aspect above, under the control of the executable computer instructions.

[0033] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, perform the interactive control method described in the first aspect above.

[0034] One beneficial effect of this disclosure is that, upon receiving a touch event, it first obtains the touch location of the touch event. If the touch location is located within one of the multiple virtual screens of the head-mounted display device, it obtains the relative position of the touch location within that virtual screen. Then, based on the relative position and the attribute information of one of the virtual screens, it re-encapsulates and distributes the touch event to that virtual screen for response. That is, when using virtual screens to implement multiple applications, the logic for touch event distribution can be modified so that the actual touch event can be distributed to a designated virtual screen for response.

[0035] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0037] Figure 1 This is a schematic diagram of the hardware configuration of a head-mounted display device according to an embodiment of the present disclosure;

[0038] Figure 2 This is one of the schematic diagrams showing a virtual screen according to an embodiment of the present disclosure;

[0039] Figure 3 This is a flowchart illustrating the interactive control method according to an embodiment of the present disclosure;

[0040] Figure 4 This is a second schematic diagram of a virtual screen display according to an embodiment of the present disclosure;

[0041] Figure 5 This is a flowchart illustrating an example of an interactive control method based on this disclosure;

[0042] Figure 6 This is a schematic block diagram of an interactive control device according to an embodiment of the present disclosure;

[0043] Figure 7 This is a schematic block diagram of a head-mounted display device according to an embodiment of the present disclosure. Detailed Implementation

[0044] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the embodiments of the present disclosure.

[0045] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0046] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0047] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0048] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0049] <Hardware Configuration>

[0050] Figure 1 This is a block diagram of the hardware configuration of a head-mounted display device 1000 according to an embodiment of the present disclosure.

[0051] like Figure 1 As shown, the head-mounted display device 1000 can be smart glasses, which can be AR glasses, or of course other devices. This disclosure does not limit the specific devices.

[0052] In one embodiment, such as Figure 1 As shown, the head-mounted display device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc.

[0053] The processor 1100 may include, but is not limited to, a central processing unit (CPU) or a microprocessor (MCU). The memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 includes, for example, various bus interfaces, such as serial bus interfaces (including USB interfaces) and parallel bus interfaces. The communication device 1400 is capable of wired or wireless communication. The display device 1500 is, for example, a liquid crystal display (LCD), an LED display, or an OLED (Organic Light-Emitting Diode) display. The input device 1600 includes, for example, a touchscreen, a keyboard, or a gamepad. The head-mounted display device 1000 can output audio information via a speaker 1700 and acquire audio information via a microphone 1800.

[0054] Those skilled in the art should understand that, although in Figure 1 The present specification shows a number of devices for a head-mounted display device 1000. However, the head-mounted display device 1000 of the embodiments described herein may involve only some of the devices, or may include other devices, which is not limited herein.

[0055] In this embodiment, the memory 1200 of the head-mounted display device 1000 is used to store instructions that control the processor 1100 to operate in order to implement or support the implementation of the interactive control method according to any embodiment. Those skilled in the art can design instructions based on the schemes disclosed in this specification. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0056] In the above description, those skilled in the art can design instructions based on the solutions provided in this disclosure. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0057] Figure 1 The head-mounted display devices shown are illustrative only and are by no means intended to limit this disclosure, its application, or its use.

[0058] Hereinafter, various embodiments and examples according to the present disclosure will be described with reference to the accompanying drawings.

[0059] <Method Implementation>

[0060] Figure 3 An embodiment of the present disclosure illustrates an interactive control method that can be implemented by a head-mounted display device, or by a control device independent of the head-mounted display device and the head-mounted display device together, or by a cloud server and the head-mounted display device together.

[0061] like Figure 3 As shown, the interactive control method of this embodiment may include the following steps S3100 to S3300:

[0062] Step S3100: Upon receiving a touch event, obtain the touch position of the touch event.

[0063] Optionally, the touch event can be a ray event sent by an interactive device, such as a mobile phone, gamepad, or mouse. For example, when interacting with a mouse, the received mouse interaction point information can be used to simulate a trigger event.

[0064] Alternatively, a touch event can be a user's (i.e., the wearer's) touch input to the desktop launcher of the head-mounted display device.

[0065] The touch location of a touch event can be the coordinates of the touch event on the desktop launcher. (See reference...) Figure 4 The touch position of the touch event is (Xt, Yt), where Xt is the horizontal coordinate of the touch position and Yt is the vertical coordinate of the touch position.

[0066] Taking head-mounted display devices, such as smart glasses or AR glasses, as an example, refer to Figure 2 As shown, the AR glasses' launcher simultaneously supports virtual screen 1 and virtual screen 2. The first application is launched on virtual screen 1, and the second application is launched on virtual screen 2. It can be understood that the AR glasses' launcher also supports a greater number of virtual screens to launch corresponding applications on those screens, thus enabling multiple application instances. When the AR glasses wearer needs to interact with an application within a virtual screen, according to the touch event handling principle, the touch event is passed through layers to the underlying driver, and then further distributed upwards, with the driver storing the corresponding touch event in the corresponding touch event node. When a touch event occurs, the Android system detects and reads the touch position (Xt, Yt).

[0067] Subsequently, proceed to step S3200: if the touch position is located within one of the multiple virtual screens of the head-mounted display device, then obtain the relative position of the touch position within that virtual screen.

[0068] The head-mounted display device includes multiple virtual screens located at different positions within the display area of ​​the head-mounted display device. (See reference...) Figure 2 As shown, the AR glasses include virtual screen 1 and virtual screen 2, and virtual screen 1 and virtual screen 2 are located in different positions on the AR glasses' desktop launcher.

[0069] Further, before executing step S3200 (if the touch position is located within one of the multiple virtual screens of the head-mounted display device, then obtaining the relative position of the touch position within that virtual screen), it is necessary to perform a step of determining whether the touch position is located within one of the multiple virtual screens of the head-mounted display device, specifically including the following steps S4100 to S4300:

[0070] Step S4100: Obtain the set mapping data; wherein, the mapping data includes attribute information of different virtual screens and their corresponding position information.

[0071] Furthermore, before performing step S4100 to obtain the set mapping data, mapping data containing the attribute information and position information of the virtual screens can be generated and stored first. Here, the interactive control method of this embodiment further includes: creating the plurality of virtual screens in the display area of ​​the head-mounted display device; recording the attribute information and position information of each virtual screen to obtain mapping data; and storing the mapping data.

[0072] Among them, the attribute information of the virtual screen can be the ID information of the virtual screen, which is used to uniquely identify the virtual screen.

[0073] The location information of the virtual screen includes the coordinates of its top-left corner and bottom-right corner. The top-left corner coordinates include both its horizontal and vertical coordinates; the bottom-right corner coordinates include both its horizontal and vertical coordinates.

[0074] Typically, after creating virtual screens on the launcher of a head-mounted display device, the information for each virtual screen needs to be recorded and saved. This saved information includes: the virtual screen's ID and its creation coordinates on the launcher. The position coordinates include the top-left corner coordinates (XL, YL) and bottom-right corner coordinates (XR, YR). XL represents the top-left horizontal coordinate, YL represents the top-left vertical coordinate, XR represents the bottom-right horizontal coordinate, and YR represents the bottom-right vertical coordinate. Each virtual screen has its own record, and the corresponding virtual screen information must be deleted when the application displayed on that screen exits.

[0075] Step S4200: Traverse the position information corresponding to each attribute information in the mapping data.

[0076] After reading the touch position of the touch event from the touch event node, before actually distributing the touch event to the corresponding application, you can first read the touch position (Xt, Yt) of the touch event, then traverse the position information corresponding to each ID information in the mapping data one by one, and compare the position information of the virtual screen read with the position information of the touch event.

[0077] Step S4300: Based on the location information corresponding to the touch position and the traversed attribute information, determine whether the touch position is located within the virtual screen indicated by the traversed attribute information.

[0078] Optionally, determining whether the touch position is within the virtual screen indicated by the traversed attribute information according to the touch position and the position information corresponding to the traversed attribute information in this step S4300 may further include: determining that the touch position is within the virtual screen indicated by the traversed attribute information when the horizontal coordinate position of the touch position is between the horizontal coordinate position of the upper left corner and the horizontal coordinate position of the lower right corner of the traversed position information, and the vertical coordinate position of the touch position is between the vertical coordinate position of the upper left corner and the vertical coordinate position of the lower right corner of the traversed position information.

[0079] Continuing with the above example, referring to Figure 4 , when the virtual screen 1 satisfies XL < Xt < XR and YL < Yt < YR, it indicates that the touch position is within the virtual screen 1. Here, XL represents the horizontal coordinate information of the upper left corner of the virtual screen 1, YL represents the vertical coordinate information of the upper left corner of the virtual screen 1, XR represents the horizontal coordinate information of the lower right corner of the virtual screen 1, and YR represents the vertical coordinate information of the lower right corner of the virtual screen 1. It can be understood that if the touch position is not within the virtual screen indicated by the traversed attribute information, it means that the touch position is not within the virtual screen range, and at this time, it is processed according to the original distribution process.

[0080] In this embodiment, the relative position of the touch position within one of the virtual screens includes: the horizontal relative position Rel_X of the touch position within one of the virtual screens and the vertical relative position Rel_Y of the touch position within one of the virtual screens. Here, obtaining the relative position of the touch position within one of the virtual screens in this step S3200 may further include: obtaining the horizontal relative position of the touch position within one of the virtual screens according to the horizontal coordinate position of the touch position and the horizontal coordinate position of the upper left corner of one of the virtual screens; obtaining the vertical relative position of the touch position within one of the virtual screens according to the vertical coordinate position of the touch position and the vertical coordinate position of the upper left corner of one of the virtual screens.

[0081] Continuing with the above example, the horizontal relative position Rel_X of the touch position within the virtual screen 1 can be calculated according to the following formula:

[0082] Rel_X = Xt – XL (1)

[0083] where, Xt represents the horizontal coordinate position of the touch position, and XL represents the horizontal coordinate position of the upper left corner of the virtual screen 1.

[0084] Furthermore, the vertical relative position Rel_Y of the touch location within virtual screen 1 can be calculated using the following formula:

[0085] Rel_Y=Yt–YL (2)

[0086] Where Yt represents the horizontal coordinate of the touch position, and YL represents the vertical coordinate of the upper left corner of the virtual screen 1.

[0087] Subsequently, in step S3300, the touch event is repackaged and distributed to one of the virtual screens for response based on the relative position and the attribute information of one of the virtual screens.

[0088] In this embodiment, the Android system supports simulating touch events using the ID information and touch position of a virtual screen. After obtaining the ID information and the relative position of the touch position within one of the virtual screens, the system sends the information to the virtual screen pointed to by the ID information according to the standard Android touch event sending mechanism.

[0089] Continuing with the example above, when the touch location is on virtual screen 1, the touch event can be redistributed to virtual screen 1 for response based on the ID information of virtual screen 1 and the relative position of the touch location within virtual screen 1.

[0090] According to embodiments of this disclosure, upon receiving a touch event, the system first obtains the touch location. If the touch location is within one of the multiple virtual screens of the head-mounted display device, the system obtains the relative position of the touch location within that virtual screen. Then, based on the relative position and the attribute information of that virtual screen, the touch event is repackaged and distributed to that virtual screen for response. That is, when using virtual screens to implement multiple applications, the logic for touch event distribution can be modified to ensure that the actual touch event is distributed to a designated virtual screen for response.

[0091] <Example>

[0092] The following example uses a head-mounted display device as AR glasses to illustrate an interactive control method. (Refer to...) Figure 2 , Figure 4 and Figure 5 The interactive control method may include the following steps:

[0093] Step 701: After creating virtual screen 1 and virtual screen 2 in AR Launcher, assign ID information to each virtual screen and record the location information of each virtual screen to establish mapping data.

[0094] Step 702, when a touch event is received, the touch event is passed layer by layer to the underlying driver, and then the touch event is delivered and distributed layer by layer upward. The corresponding touch event is stored in the corresponding touch event node from the driver, and the Android system reads the touch position of the touch event.

[0095] Step 703, traverse the position information corresponding to each attribute information in the mapping data. According to the touch position and the position information corresponding to the traversed attribute information, determine whether the touch position (Xt, Yt) is within the virtual screen indicated by the traversed attribute information. When virtual screen 1 satisfies XL < Xt < XR and YL < Yt < YR, it is determined that the touch position is within virtual screen 1, and step 704 is executed. Otherwise, step S706 is executed.

[0096] Step 704, when the touch position is within virtual screen 1, obtain the relative position of the touch position within virtual screen 1.

[0097] Step 705, according to the relative position and the attribute information of virtual screen 1, repackage and distribute the touch event to virtual screen 1 for response, and the process ends.

[0098] Step 706, when the touch position is not within the virtual screen indicated by each traversed attribute information, it indicates that the touch position is not within the virtual screen range, and then it is distributed according to the original process, and the process ends.

[0099] <Device Embodiment>

[0100] Figure 6 It is a schematic diagram of the principle of an interactive control device according to an embodiment. Refer to Figure 6 As shown, the device 600 includes a receiving module 610, a first obtaining module 620, a second obtaining module 630, and a packaging module 640.

[0101] The receiving module 610 is configured to receive a touch event;

[0102] The first obtaining module 620 is configured to obtain the touch position of the touch event;

[0103] The second obtaining module 630 is configured to, if the touch position is within one of the multiple virtual screens of the head-mounted display device, obtain the relative position of the touch position within the one virtual screen. The head-mounted display device includes multiple virtual screens, and the multiple virtual screens are located at different positions in the display area of the head-mounted display device;

[0104] The encapsulation module 640 is used to re-encapsulate and distribute the touch event to one of the virtual screens for response based on the relative position and the attribute information of one of the virtual screens.

[0105] In one embodiment, the device 600 further includes a determining module (not shown), configured to:

[0106] Obtain the set mapping data; wherein, the mapping data includes attribute information of different virtual screens and their corresponding position information;

[0107] Traverse the position information corresponding to each attribute information in the mapping data;

[0108] Based on the touch position and the position information corresponding to the traversed attribute information, determine whether the touch position is located within the virtual screen indicated by the traversed attribute information.

[0109] In one embodiment, the determining module is specifically used for:

[0110] If the horizontal coordinate position of the touch position is located between the upper left and lower right horizontal coordinate positions of the traversed location information, and the vertical coordinate position of the touch position is located between the upper left and lower right vertical coordinate positions of the traversed location information, then the touch position is determined to be within the virtual screen indicated by the traversed attribute information.

[0111] In one embodiment, the relative position of the touch position within one of the virtual screens includes: the horizontal relative position of the touch position within one of the virtual screens, and the vertical relative position of the touch position within one of the virtual screens. The second acquisition module 630 is used for:

[0112] Based on the horizontal coordinates of the touch location and the horizontal coordinates of the top left corner of one of the virtual screens, the horizontal relative position of the touch location within one of the virtual screens is obtained;

[0113] Based on the vertical coordinates of the touch position and the vertical coordinates of the top left corner of one of the virtual screens, the vertical relative position of the touch position within one of the virtual screens is obtained.

[0114] In one embodiment, the device further includes a storage module (not shown).

[0115] A storage module is used to create the plurality of virtual screens in the display area of ​​the head-mounted display device;

[0116] Record the attribute and position information of each virtual screen to obtain mapping data;

[0117] Store the mapping data;

[0118] The location information of the virtual screen includes: the coordinates of the upper left corner of the virtual screen and the coordinates of the lower right corner of the virtual screen;

[0119] The coordinates of the upper left corner of the virtual screen include: the horizontal coordinates of the upper left corner of the virtual screen and the vertical coordinates of the upper left corner of the virtual screen.

[0120] The coordinates of the lower right corner of the virtual screen include: the horizontal coordinates of the lower right corner of the virtual screen and the vertical coordinates of the lower right corner of the virtual screen.

[0121] According to embodiments of this disclosure, upon receiving a touch event, the system first obtains the touch location. If the touch location is within one of the multiple virtual screens of the head-mounted display device, the system obtains the relative position of the touch location within that virtual screen. Then, based on the relative position and the attribute information of that virtual screen, the touch event is repackaged and distributed to that virtual screen for response. That is, when using virtual screens to implement multiple applications, the logic for touch event distribution can be modified to ensure that the actual touch event is distributed to a designated virtual screen for response.

[0122] <Equipment Example>

[0123] Figure 7 This is a schematic diagram of the hardware structure of a head-mounted display device according to one embodiment. Figure 7 As shown, the head-mounted display device 700 includes a processor 710 and a memory 720.

[0124] The memory 720 can be used to store executable computer instructions.

[0125] The processor 710 can be used to execute the interactive control method according to the method embodiments of this disclosure, under the control of the executable computer instructions.

[0126] The head-mounted display device 700 can be as follows: Figure 1 The head-mounted display device 1000 shown may also be a device with other hardware structures, which are not limited here.

[0127] In another embodiment, the head-mounted display device 700 may include the above-mentioned interactive control device 600.

[0128] In one embodiment, each module of the above-mentioned interactive control device 600 can be implemented by the processor 710 running computer instructions stored in the memory 720.

[0129] Computer-readable storage media

[0130] This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the interactive control method provided in this disclosure.

[0131] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0132] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0133] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0134] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0135] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0137] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0139] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. An interactive control method, characterized in that, The method includes: Upon receiving a touch event, the touch location of the touch event is obtained; If the touch position is located within one of the multiple virtual screens of the head-mounted display device, then the relative position of the touch position within that virtual screen is obtained, wherein the head-mounted display device includes multiple virtual screens, and the multiple virtual screens are located at different positions in the display area of ​​the head-mounted display device; Based on the relative position and the attribute information of one of the virtual screens, the touch event is re-encapsulated and distributed to one of the virtual screens for response; The method further includes: acquiring set mapping data; wherein the mapping data includes attribute information of different virtual screens and their corresponding position information; traversing the position information corresponding to each attribute information in the mapping data; and determining whether the touch position is located within the virtual screen indicated by the traversed attribute information based on the touch position and the position information corresponding to the traversed attribute information.

2. The method according to claim 1, characterized in that, The step of determining whether the touch position is located within the virtual screen indicated by the traversed attribute information based on the touch position and the position information corresponding to the traversed attribute information includes: If the horizontal coordinate position of the touch position is located between the upper left and lower right horizontal coordinate positions of the traversed location information, and the vertical coordinate position of the touch position is located between the upper left and lower right vertical coordinate positions of the traversed location information, then the touch position is determined to be within the virtual screen indicated by the traversed attribute information.

3. The method according to claim 1, characterized in that, The relative position of the touch position within one of the virtual screens includes: the horizontal relative position of the touch position within one of the virtual screens, and the vertical relative position of the touch position within one of the virtual screens. The step of obtaining the relative position of the touch location within one of the virtual screens includes: Based on the horizontal coordinates of the touch location and the horizontal coordinates of the top left corner of one of the virtual screens, the horizontal relative position of the touch location within one of the virtual screens is obtained; Based on the vertical coordinates of the touch position and the vertical coordinates of the top left corner of one of the virtual screens, the vertical relative position of the touch position within one of the virtual screens is obtained.

4. The method according to claim 1, characterized in that, The method further includes: The plurality of virtual screens are created in the display area of ​​the head-mounted display device; Record the attribute and position information of each virtual screen to obtain mapping data; Store the mapping data; The location information of the virtual screen includes: the coordinates of the upper left corner of the virtual screen and the coordinates of the lower right corner of the virtual screen; The coordinates of the upper left corner of the virtual screen include: the horizontal coordinates of the upper left corner of the virtual screen and the vertical coordinates of the upper left corner of the virtual screen. The coordinates of the lower right corner of the virtual screen include: the horizontal coordinates of the lower right corner of the virtual screen and the vertical coordinates of the lower right corner of the virtual screen.

5. An interactive control device, the device comprising: The receiving module is used to receive touch events; The first acquisition module is used to acquire the touch position of the touch event; The second acquisition module is used to acquire the relative position of the touch position within one of the multiple virtual screens of the head-mounted display device if the touch position is located within one of the multiple virtual screens of the head-mounted display device. The head-mounted display device includes multiple virtual screens, and the multiple virtual screens are located at different positions in the display area of ​​the head-mounted display device. An encapsulation module is used to re-encapsulate and distribute the touch event to one of the virtual screens for response based on the relative position and the attribute information of one of the virtual screens; The device further includes a determining module, configured to: acquire set mapping data; wherein the mapping data includes attribute information of different virtual screens and their corresponding position information; traverse the position information corresponding to each attribute information in the mapping data; and determine whether the touch position is located within the virtual screen indicated by the traversed attribute information based on the touch position and the position information corresponding to the traversed attribute information.

6. The apparatus according to claim 5, characterized in that, The determining module is specifically used for: If the horizontal coordinate position of the touch position is located between the upper left and lower right horizontal coordinate positions of the traversed location information, and the vertical coordinate position of the touch position is located between the upper left and lower right vertical coordinate positions of the traversed location information, then the touch position is determined to be within the virtual screen indicated by the traversed attribute information.

7. A head-mounted display device, characterized in that, The head-mounted display device includes: Memory is used to store executable computer instructions; A processor configured to execute the interactive control method according to any one of claims 1-4, under the control of the executable computer instructions.

8. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, perform the interactive control method according to any one of claims 1-4.

Images