Display method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a display method and device, equipment and a storage medium. The method includes acquiring a pose of the manipulation device. And displaying a virtual interface of the control equipment in a first display area corresponding to the control equipment on the head-mounted display equipment according to the pose of the control equipment. In response to detecting the touch of the object on the manipulation device, a touch location of the object on the manipulation device is determined. And according to the touch position, displaying a first indication in a virtual interface of the control equipment to indicate the touch position of the object. In this way, convenience is provided for a user to operate the manipulation device, the interaction mode of the manipulation device is expanded, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, apparatus, device and computer readable storage medium for display. BACKGROUND

[0002] Extended Reality (XR) is widely researched and applied. XR combines hardware devices and various technical means to fuse virtual content and real scenes, and provides users with unique sensory experiences. XR includes, for example, Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0003] Users can access virtual scenes by wearing head-mounted display devices to obtain immersive interactive experiences. However, current extended reality technology still has some problems that affect the participation experience of users. SUMMARY

[0004] In a first aspect of the present disclosure, a method for display is provided. The method includes: obtaining a pose of a manipulation device; displaying, on a head-mounted display device, a virtual interface of the manipulation device in a first display area corresponding to the manipulation device according to the pose of the manipulation device; in response to detecting a touch of an object on the manipulation device, determining a touch position of the object on the manipulation device; and displaying a first indication in the virtual interface of the manipulation device according to the touch position to indicate the touch position of the object.

[0005] In a second aspect of the present disclosure, an apparatus for display is provided. The apparatus includes: an obtaining module configured to obtain a pose of a manipulation device; a first display module configured to display, on a head-mounted display device, a virtual interface of the manipulation device in a first display area corresponding to the manipulation device according to the pose of the manipulation device; and a determining module configured to determine a touch position of an object on the manipulation device in response to detecting a touch of the object on the manipulation device, wherein the display module is further configured to display a first indication in the virtual interface of the manipulation device according to the touch position to indicate the touch position of the object.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A schematic diagram illustrating a display process according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrates an example process for detecting the hovering position of an object on a manipulating device according to some embodiments of the present disclosure;

[0013] Figure 4 A schematic diagram illustrating an example process for displaying the hovering position of an object on a manipulating device according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic diagram illustrating a process for determining the hovering position of an object on a manipulating device according to some embodiments of the present disclosure is shown;

[0015] Figure 6 Block diagrams of a display apparatus according to some embodiments of the present disclosure are shown; and

[0016] Figure 7 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0017] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0018] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0019] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0020] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0024] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0025] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. Environment 100 can be a physical scene, i.e., environment 100 can be an example of a real-world scene. In environment 100, user 110 can perform interactive operations by wearing at least one head-mounted display device 120. User 110 can also interact with manipulator 140 by operating a handheld control device 140. For example, user 110 can operate manipulator 140 by touching manipulator 140 with an object 170 (e.g., a finger, stylus, etc.). In some embodiments, manipulator 140 can be an external power supply that can be wired to head-mounted display device 120. It is understood that the control device can also be other external devices, such as a touchpad, mobile communication device, or external communication module, and the control device can have a regular plane or curved surface for interaction with objects. In some embodiments, head-mounted display device 120 can communicate with manipulator 140 and server 130 to achieve interaction with user 110 in a collaborative manner.

[0027] In some embodiments, the head-mounted display device 120 may include devices for acquiring images, such as cameras, infrared cameras, depth cameras, and other image acquisition devices, to capture images of the physical space. One or more image acquisition devices may be respectively mounted on the head-mounted display device 120 or deployed outside the head-mounted display device. For example, an image acquisition device may be mounted on the head-mounted display device 120 to acquire images over a wider spatial range based on a higher spatial position. If an infrared camera is mounted on the head-mounted display device 120, correspondingly, at least one infrared light source may be mounted on the control device 140 so that the infrared camera can capture the infrared light emitted by the infrared light source on the control device 140, thereby calculating the pose of the control device 140 based on the light spot corresponding to the light source in the obtained image, and recognizing the user 110's operation commands based on the obtained infrared image of the environment. It is understood that, depending on the different image acquisition devices, various types of images such as RGB images and depth images may also be obtained. Based on one or more of these images, the pose of the control device 140 can be calculated, and the object 170 or operation commands can be recognized.

[0028] In other scenarios, the image acquisition device used for image acquisition can be independent of the head-mounted display device 120 (e.g., independent of the head-mounted display device 120). Such an image acquisition device can be deployed independently for image perception of manipulating devices 140 and other entities in the environment. Exemplarily, other entities can be furniture, furnishings, etc. in the environment.

[0029] In some embodiments, sensor devices may also be deployed in the head-mounted display device 120, the control device 140, and / or the environment to collect information about the environment, including information related to the control device 140. The sensor devices may include image sensor devices and sensors that sense other physical quantities such as temperature, light, color, speed, and acceleration.

[0030] In some embodiments, user 110 may communicate with server 130 using multiple head-mounted display devices 120 to process data acquired through head-mounted display devices 120 and send data to head-mounted display devices 120. It should be understood that in some cases, head-mounted display devices 120 may not communicate with server 130 (or, in other words, the environment may exemplary exclude server 130). Furthermore, head-mounted display devices 120 can interact with devices such as manipulation devices 140. Specifically, interaction with other head-mounted display devices can be achieved through a controller within head-mounted display devices 120. It should be understood that the controller of head-mounted display devices 120 may be located within other head-mounted display devices or may exist as a standalone control device.

[0031] In environment 100, user 110 can use head-mounted display device 120 to access and display virtual environment 150. Virtual environment 150 may exemplary be a space. In some embodiments, virtual content 160 may be generated in virtual environment 150 to enable user 110 to have an immersive interactive experience in virtual environment 150.

[0032] Server 130 may be a separate device capable of communicating with head-mounted display device 120 and / or other image capture devices, such as a server 130 for image or data processing, a computing node, etc., or it may be integrated with head-mounted display device 120 and / or other image capture devices. In some embodiments, server 130 may be implemented as head-mounted display device 120, that is, in this case, head-mounted display device 120 may implement all the functions of server 130. It should be understood that the above description of server 130 is merely exemplary and not limiting, and server 130 may be implemented as a device of various forms, structures, or categories, and the embodiments of this disclosure are not limited in this regard.

[0033] For example, server 130 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. Terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Server devices may be independent physical servers 130, server clusters or distributed systems composed of multiple physical servers 130, or cloud servers 130 providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server devices may include, for example, computing systems / servers 130, such as mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0034] like Figure 1 As shown, the head-mounted display device 120 can provide a virtual environment 150, including virtual content 160, to a user 110. In some embodiments, the server 130 can run scene instances and utilize computing resources to render them to generate virtual scenes, which the user 110 can access through the head-mounted display device 120. Alternatively, the scene instance is run by the head-mounted display device 120. Such virtual scenes may include, but are not limited to, games (such as bullet hell games), virtual reality scenes, head-mounted display device user interfaces (UIs), etc.

[0035] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0036] As briefly mentioned earlier, users can achieve an immersive experience in a virtual environment by wearing a head-mounted display (HUD). When operating the HUD, users can input commands to obtain corresponding services. For example, the HUD can display virtual buttons or a virtual keyboard. Users can trigger the corresponding function of the virtual buttons by placing their fingers on the areas corresponding to the virtual buttons.

[0037] Currently, users mostly input commands to head-mounted displays through control devices. However, while wearing a head-mounted display, the user's field of vision is obstructed. Users cannot accurately determine the touch location, causing inconvenience in operating the control device.

[0038] Embodiments of this disclosure provide a display scheme. According to various embodiments of this disclosure, the pose of a manipulation device is obtained. Based on the pose of the manipulation device, a virtual interface of the manipulation device is displayed on a head-mounted display device in a first display area corresponding to the interface of the manipulation device. In response to detecting a touch of an object on the manipulation device, the touch position of the object on the manipulation device is determined. Based on the touch position, a first indication is displayed in the virtual interface of the manipulation device to indicate the touch position of the object.

[0039] In this way, on the one hand, during user interaction with the control device, the virtual interface of the control device is displayed in the display area of ​​the head-mounted display, and the touch position of the object is indicated in the virtual interface. This allows users to accurately determine their touch position on the control device, providing convenience for user operation and better interactive feedback. On the other hand, by displaying the virtual interface of the control device, users can interact with the control device through the virtual interface, expanding the interaction methods of the control device and enabling users to interact with the control device more flexibly. Therefore, the user experience is improved.

[0040] Figure 2 A flowchart of a process 200 for displaying the relative position of an object and a manipulation device according to some embodiments of the present disclosure is shown. Process 200 can be implemented at a head-mounted display device 120 and / or a server 130. Reference will be made below. Figure 1 The following explanation will be based on the example of process 200 being executed by the head-mounted display device 120.

[0041] In block 210, the pose of the manipulation device 140 is acquired. In some embodiments, the head-mounted display device 120 may first acquire the pose of the manipulation device 140. The pose of the manipulation device 140 may include the position and / or orientation of the manipulation device 140. The position of the manipulation device 140 may include the absolute position (e.g., coordinates in the world coordinate system) and / or the relative position (e.g., coordinates in the coordinate system of the head-mounted display device 120). The orientation of the manipulation device 140 may include the horizontal and / or vertical state, rotation state, tilt angle, etc., of the manipulation device 140. The pose of the manipulation device 140 may also include the movement trajectory of the manipulation device 140.

[0042] In some embodiments, the pose of the manipulation device 140 is determined based on at least one of the following: an image of the manipulation device 140 captured by an image capture device, and data about the manipulation device 140 detected by a positioning device. In some embodiments, the position of the manipulation device 140 can be determined based on image information captured by an image capture device (e.g., a camera, image sensor, etc.). As an example, the pose of the manipulation device 140 in space can be determined based on image information captured by an image capture device and known shape feature data of the image capture device. Alternatively or additionally, the image captured by the image capture device can be combined with data sensed by other sensors (e.g., a gyroscope, accelerometer, etc. disposed in the manipulation device 140) to improve the accuracy and stability of pose detection.

[0043] In some embodiments, the pose of the control device 140 can be determined based on positioning information obtained from positioning devices pre-positioned at the control device 140 and / or arranged in the environment. In some other embodiments, the position of the control device 140 can be determined based on infrared light emitted by an infrared light source disposed on the control device 140. For example, infrared light information emitted by an infrared light source on the control device 140 can be captured by an infrared camera disposed in the head-mounted display device 120 and / or arranged in the environment, and the pose of the control device 140 can be determined based on the captured infrared light information.

[0044] In frame 220, based on the pose of the control device 140, a virtual interface of the control device 140 is displayed on the head-mounted display device 120 in a first display area corresponding to the control device 140. The first display area can be the display area on the head-mounted display device 120 corresponding to the control device. For example, the head-mounted display device 120 determines the position and tilt angle of the control device 140 based on the acquired pose. Subsequently, the virtual interface of the control device 140 is displayed in the first display area based on the position and tilt angle of the control device 140. The control device 140 may have a user interface, or it may have an operation keyboard instead of a user interface.

[0045] In some embodiments, the real-world image captured by the image capture device can be processed and displayed on the head-mounted device to create a perspective view. The image of the control device 140 in the real-world environment will also be displayed on the head-mounted device. Virtual content (e.g., a virtual keyboard and / or virtual images) can then be displayed in the first display area corresponding to the control device 140. This virtual content can cover a portion of the control device 140 (typically a regular plane or curved surface of the control device 140) and move with the control device 140, serving as a virtual interface for the control device 140. In some embodiments, the virtual keyboard and / or virtual images can be projected as the virtual interface of the control device 140. Thus, the control device 140, which does not have a display function, can acquire a display function through the display of a virtual interface in an extended display. Furthermore, the virtual interface can be changed based on different settings, greatly expanding the interaction methods of the control device and reducing the hardware requirements of the control device 140.

[0046] In block 230, in response to detecting a touch of object 170 on manipulation device 140, the touch position of object 170 on manipulation device 140 is determined. For example, user 110 can touch any designated location on manipulation device 140 with object 170 (e.g., a finger, stylus, etc.). In some embodiments, manipulation device 140 can detect the touch of object 170 and determine the touch position. In some other embodiments, head-mounted display device 120 can acquire information related to object 170 and manipulation device 140 through sensor devices disposed in head-mounted display device 120 and / or the environment, and determine the touch position of object 170 on manipulation device 140 based on the acquired information.

[0047] In some embodiments, the manipulation device 140 can generate a corresponding signal based on the touch location after detecting a touch of an object and send it to the head-mounted display device 120. Alternatively or additionally, the head-mounted display device 120 can obtain the operation command from the user 110 based on the determined object touch location. Accordingly, the head-mounted display device 120 can provide services according to the operation command from the user 110. For example, the user 110 can trigger a corresponding virtual key on the virtual keyboard displayed in the first display area of ​​the head-mounted display device 120 by touching a certain area on the manipulation device 140.

[0048] In box 240, a first indication is displayed in the virtual interface of the manipulation device 140 according to the touch position to indicate the touch position of the object 170. In some embodiments, the head-mounted display device 120 may display the first indication in the virtual interface of the manipulation device 140 according to the touch position of the object 170 on the manipulation device 140 to indicate the touch position of the object 170. In some embodiments, a mapped position of the touch position in a first display area is determined, and the first indication may indicate the mapped position.

[0049] For example, the first indication may be in the form of an icon. For instance, a specific icon may be displayed in the area corresponding to the touch location within the first display area. Alternatively or additionally, the first indication may be a combination of icon and text. In some embodiments, an icon (i.e., the first indication) indicating the touch location of the user 110 may be displayed in the virtual content (i.e., the virtual interface of the manipulation device 140) displayed in the first display area of ​​the head-mounted display device 120. It is understood that the first indication may be reflected through changes to the virtual interface of the manipulation device 140 displayed in the first display area. For example, if the virtual content is a keyboard, a key may be displayed as if it has been pressed based on a touch signal. The first indication may also be overlaid on the content of the virtual interface within the first display area without changing the virtual interface content.

[0050] By indicating the touch position of object 170 in the virtual interface of the control device 140, user 110 can conveniently, quickly, intuitively and accurately determine the touch position of object 170 on the control device 170, thereby enabling user 110 to interact with the control device 170 more efficiently and improving user experience.

[0051] In some embodiments, process 200 may further include: determining the hovering position of the object 170 on the manipulation device 140 in response to detecting hovering of the object 170 on the manipulation device 140. Based on the hovering position of the object 170 on the manipulation device 140, displaying a second indication in a first display area on the head-mounted display device 120 to indicate the hovering position of the object. Similar to the first indication, the second indication may be in the form of an icon, a combination of icon and text, or any other suitable form. It is understood that the second indication may be reflected through changes in the virtual interface of the manipulation device 140 displayed in the first display area, or it may be displayed overlaid in the first display area without changing the content of the virtual interface.

[0052] By detecting and indicating the hovering position of object 170 on control device 140 to user 110, user 110 can accurately know the relative position of object 170 and control device 140, thereby enabling more accurate input of commands to control device 140. This provides user 110 with faster, more accurate, and more efficient interactive feedback.

[0053] In some embodiments, to determine the hovering position of object 170 on manipulation device 140, at least one of the object's position and pose can be acquired, and the hovering position of object 170 on manipulation device 140 is determined based on the pose of manipulation device 140 and at least one of the object's position and pose. In some embodiments, at least one of the object's position and pose is determined based on at least one of the following: an image of object 170 captured by an image capture device, and the position of object 170 relative to manipulation device 140 detected by manipulation device 140.

[0054] As an example, a camera and a sensor device located in the head-mounted display device 120 and / or the environment can collect information related to the object 170 and the manipulation device 140, and determine at least one of the position and pose of the object 170 and / or the pose of the manipulation device 140 based on the collected information.

[0055] The hover position of object 170 on the control device 140 can indicate the position of object 170 before it comes into contact with the control device 140. For example, if the control device 140 is a tablet device and object 170 is user 110's finger, then the hover position indicates the position of the fingertip after user 110 extends their finger, provided that the distance between the fingertip and the tablet screen does not exceed a preset distance. The hover position can be represented by the coordinate information of the projection of object 170 onto the plane associated with the control device 140 and / or the distance between object 170 and the control device 140.

[0056] In some embodiments, at least one of the pose of the manipulation device 140 and the position and / or pose of the object 170 can be determined separately. Based on the pose of the manipulation device 140 and at least one of the position and / or pose of the object 170, the hovering position of the object 170 within the manipulation device 140 is determined. Regarding the pose of the manipulation device 140, any of the methods discussed above for determining the pose of the manipulation device 140 can be used, and will not be repeated here.

[0057] In some embodiments, the position and / or pose of object 170 can be determined based on image information captured by the image capturing device and known shape feature data of the image capturing device. In some embodiments, the position and / or pose of object 170 can be determined based on positioning information sent by a positioning device pre-set at the manipulation device 140 or object 170.

[0058] In some embodiments, the camera can directly determine the relative position of the manipulation device 140 and the object 170, and determine the hovering position of the object 170 on the manipulation device 140 based on the relative position. For example, a touch screen (e.g., a pressure sensor) or a touchpad with an electric field at the manipulation device 140 can be used to determine the relative position of the object 170 and the manipulation device 140. Figure 3 A schematic diagram of an example process 300 for detecting the hovering position of an object on a manipulating device, according to some embodiments of the present disclosure, is shown. Figure 3 As shown, the hovering position of object 170 can be detected by a touchpad in the control device 140 that can generate an electric field. If user 110 interacts with control device 140 using object 170, the position of object 170 relative to control device 140 (i.e., the hovering position of object 170 on control device 140) can be determined based on the electric field change information generated on the touchpad surface.

[0059] In some embodiments, information about the position and / or pose of object 170 can be obtained through a multimodal device. For example, the position and / or pose information of 170 can be obtained by manipulating a touchscreen at device 140, a touchpad with an electric field, and a head-mounted display device 120 and / or an image capture device arranged in the environment. In some embodiments, different weights can be assigned to these different pieces of information to further improve the positioning accuracy of object 170.

[0060] In some embodiments, the mapped position of object 170 in the first display area can be determined based on the hovering position of object 170 on manipulating device 140. Subsequently, a second indication can be displayed in the first display area on head-mounted display device 120 to indicate the mapped position. Figure 4 A schematic diagram illustrating a process 400 of displaying the hover position of an object on a manipulating device according to some embodiments of the present disclosure is shown. Figure 4 As shown, if user 110 operates control device 140 through object 170, before object 170 touches control device 140, an indication 410-1 (i.e., second indication) corresponding to the hovering position of object 170 is presented in the virtual environment 150 displayed by head-mounted display device 120.

[0061] In some embodiments, in response to detecting a change in the hovering position of object 170 on the manipulation device, a third indication may also be displayed in a first display area on the head-mounted display device 120 to indicate the updated hovering position of object 170. For example, as Figure 4 As shown, when object 170 moves relative to the manipulation device 140, the hovering position of object 170 on the manipulation device 140 changes. Correspondingly, the mapping position changes, and the mapping position indicator 410-1 changes to indicator 410-2 (i.e., the third indicator). This allows for timely indication of the change in the hovering position of object 170 to the user 110, further improving the interaction efficiency between the user 110 and the manipulation device 140, thereby enhancing the user experience. Furthermore, the user 110 can correct the hovering position by clicking, further improving the accuracy and efficiency of user operations.

[0062] In some embodiments, in addition to displaying in a first display area on the head-mounted display device 120, display can also be made in a second display area on the head-mounted display device 120. The second display area may be a display area on the display of the head-mounted display device that corresponds to the area directly in front of the user 110's eyes. A virtual environment 150 including virtual content 160 can be provided to the user 110 through the second display area, allowing the user 110 to have an immersive interactive experience in the virtual environment 150.

[0063] In some embodiments, a second indication indicating the hovering position of object 170 on manipulation device 140 may be displayed in a second display area on head-mounted display device 120. For example, after determining the hovering position of object 170 on manipulation device 140 in response to detecting hovering of object 170 on manipulation device 140, a second indication may be displayed in the second display area on head-mounted display device to indicate the hovering position of object 170 based on the hovering position of object 170 on manipulation device 140. Any of the methods discussed above for detecting hovering of object 170 on manipulation device 140 and determining and indicating the hovering position may be used, and will not be repeated here. By displaying the hovering pose of object 170 in a second display area, for example, in front of the user's eyes, user 110 can see the projected landing point of object 170 in the second display area before the object touches manipulation device 140 (e.g., before a finger presses against the display screen of manipulation device 140), thereby enabling more reliable and accurate click operations.

[0064] In some embodiments, the image displayed on the virtual interface of the control device 140 presented in the first display area of ​​the head-mounted display device 120 may be the same as the image displayed in the second display area of ​​the head-mounted display device 120. By displaying the same image and / or content in the first and second display areas, the user 110 can directly see the same image and / or content on the control device 140 through the head-mounted display device 120. Therefore, the user 110 can more conveniently and intuitively issue operation commands by operating the control device 140 to trigger corresponding functions and / or obtain the required services, thereby further improving the user 110's operational efficiency. The second display area of ​​the head-mounted display device 120 may be a display area displayed at a fixed position on the head-mounted display screen, or the second display area may be anchored to a preset position in the real environment and displayed through the head-mounted display device. Therefore, even if the first display area changes with the position of the control device, or is sometimes not displayed, the user can still conveniently and intuitively view the virtual interface of the control device 140 through the stably displayed second display area.

[0065] In some embodiments, in response to detecting a change in the hovering position of the object on the manipulation device 140, a third indication may be displayed in a second display area on the head-mounted display device 120 to indicate the updated hovering position of the object. Any of the methods discussed above for detecting and indicating changes in the hovering position of the object 170 on the manipulation device 140 may be employed, and will not be repeated here.

[0066] The following is for reference. Figure 5 An example procedure for determining the hovering position of object 170 on manipulating device 140 is discussed. For example... Figure 5As shown, in box 510, the head-mounted camera (i.e., the camera on the head-mounted display device 120) can identify the manipulation device 140. In box 520, the inertial measurement unit (IMU) information of the manipulation device 140 can be acquired. In box 550, based on the acquired IMU information of the manipulation device 140 and the recognition result of the head-mounted camera, the manipulation device 140 is positioned to determine the pose of the manipulation device 140. In box 530, the head-mounted camera also identifies the joint information of the hand (for example, object 170) (e.g., the (x, y, z) coordinate information of the hand joints). In box 540, the electric field sensing of the touchpad of the manipulation device 140 is detected. In box 560, based on the acquired electric field sensing information and the joint information of the hand, the position information of the hand on the manipulation device 140 is mapped onto the screen of the head-mounted display device 120 (e.g., including a first display area and / or a second display area) to indicate the hovering position of the hand. In box 580, the head-mounted camera acquires an environmental image of the surrounding environment. In box 590, the environmental image acquired by the head-mounted camera is displayed on a head-mounted display used to show the position information of the hand on the manipulation device 140.

[0067] In this way, the position of the user's hand can be determined before the user's hand touches the screen of the control device 140. By displaying the hovering position of the user's hand on the screen of the head-mounted display device 120 or the display system, the user can know the position of their hand before clicking, thereby improving the user's input accuracy and efficiency.

[0068] It should be understood that the embodiments of this disclosure can be applied to various virtual input scenarios, such as games and text input scenarios. The scope of this disclosure is not limited in this respect.

[0069] Figure 6 A block diagram of a display apparatus 600 according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in a head-mounted display device 120. Various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0070] As shown in the figure, device 600 includes an acquisition module 610 configured to acquire the pose of a manipulating device. Device 600 also includes a display module 620 configured to display a virtual interface of the manipulating device on a head-mounted display device corresponding to the manipulating device, based on the pose of the manipulating device. Device 600 further includes a determination module 630 configured to determine the touch position of an object on the manipulating device in response to detecting a touch on the manipulating device. The display module 620 is further configured to display a first indication in the virtual interface of the manipulating device based on the touch position to indicate the touch position of the object.

[0071] In some embodiments, the acquisition module 610 is further configured to determine the pose of the manipulating device based on at least one of the following: an image of the manipulating device captured by an image capturing device, and data about the manipulating device detected by a positioning device.

[0072] In some embodiments, the display module 620 is further configured to determine the hovering position of the object on the manipulation device in response to detecting the hovering of the object on the manipulation device; and to display a second indication in a first display area on the head-mounted display device to indicate the hovering position of the object based on the hovering position of the object on the manipulation device.

[0073] In some embodiments, the display module 620 is further configured to determine the hovering position of the object on the manipulation device in response to detecting the hovering of the object on the manipulation device; and to display a second indication in a second display area on the head-mounted display device to indicate the hovering position of the object based on the hovering position of the object on the manipulation device.

[0074] In some embodiments, the image displayed on the virtual interface of the operating device is the same as the image displayed in the second display area.

[0075] In some embodiments, the display module 620 is further configured to determine the mapped position of the object in the display area based on the hovering position of the object on the manipulation device; and to display a second indication in the display area on the head-mounted display device to indicate the mapped position.

[0076] In some embodiments, the display module 620 is further configured to acquire at least one of the position and pose of the object; and to determine the hovering position of the object on the manipulation device based on the pose of the manipulation device and at least one of the position and pose of the object.

[0077] In some embodiments, the display module 620 is further configured to be an image of an object captured by an image capturing device, and the position of the object relative to the manipulating device detected by the manipulating device.

[0078] In some embodiments, the device 600 further includes an update module configured to display a third indication in a display area on a head-mounted display device in response to detecting a change in the hovering position of an object on a manipulating device, to indicate the updated hovering position of the object.

[0079] In some embodiments, the control device is an external power supply that is wired to the head-mounted display device.

[0080] Figure 7 A block diagram is shown illustrating an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 7The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to achieve Figure 1 Electronic devices 110.

[0081] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0082] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0083] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0084] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0085] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0086] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0087] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0089] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0091] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for display, comprising: obtaining a pose of a manipulation device; displaying, on a head-mounted display device, a virtual interface of the manipulation device in a first display area corresponding to the manipulation device according to the pose of the manipulation device; in response to detecting a touch of an object on the manipulation device, determining a touch position of the object on the manipulation device; and displaying a first indication in the virtual interface of the manipulation device according to the touch position to indicate the touch position of the object.

2. The method of claim 1, further comprising: in response to detecting a hover of the object on the manipulation device, determining a hover position of the object on the manipulation device; and displaying a second indication in the first display area on the head-mounted display device according to the hover position of the object on the manipulation device to indicate the hover position of the object.

3. The method of claim 1, further comprising: in response to detecting a hover of the object on the manipulation device, determining a hover position of the object on the manipulation device; and displaying a second indication in a second display area on the head-mounted display device according to the hover position of the object on the manipulation device to indicate the hover position of the object.

4. The method of claim 3, wherein an image displayed on the virtual interface of the manipulation device is the same as an image displayed in the second display area.

5. The method of claim 2 or 3, wherein displaying the second indication in the display area on the head-mounted display device comprises: determining a mapped position of the object in the display area based on the hover position of the object on the manipulation device; and displaying the second indication in the display area on the head-mounted display device to indicate the mapped position.

6. The method of claim 2 or 3, wherein the determining the hover position of the object on the manipulation device comprises: obtaining at least one of a position and a pose of the object; and determining the hover position of the object on the manipulation device based on the pose of the manipulation device and the at least one of the position and the pose of the object.

7. The method of claim 6, wherein the at least one of the position and the pose of the object is determined based on at least one of: an image of the object captured by an image capture device, a position of the object relative to the manipulation device detected by the manipulation device.

8. The method of claim 2 or 3, further comprising: in response to detecting a change in the hover position of the object on the manipulation device, displaying a third indication in the display area on the head-mounted display device to indicate an updated hover position of the object.

9. The method of claim 1, wherein the pose of the manipulation device is determined based on at least one of: an image of the manipulation device captured by an image capture device, data about the manipulation device detected by a positioning device. ​ ​ ​ ​ ​ 10. The method of claim 1, wherein the manipulation device is an external power source that is connected to the head-mounted display device in a wired manner.

11. An apparatus for display, comprising: an obtaining module configured to obtain a pose of a manipulation device; a display module configured to display a virtual interface of the manipulation device in a first display area corresponding to the manipulation device on a head-mounted display device according to the pose of the manipulation device; and a determining module configured to determine a touch position of an object on the manipulation device in response to detecting a touch of the object on the manipulation device, wherein the display module is further configured to display a first indication in the virtual interface of the manipulation device according to the touch position to indicate the touch position of the object.

12. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-10.

13. A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-10.

14. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1-10. ​

Images