Display method and apparatus for head-mounted display device, device and product

Abstract

A display method and apparatus for a head-mounted display device, an electronic device, and a product. The method comprises: in response to a first region of an interaction device (110) of a head-mounted display device intersecting a second region of a physical object (130) in a physical environment, acquiring information of the physical object (130); on the basis of the information of the physical object (130), determining a see-through region (120) that is displayed in the head-mounted display device in a see-through way; and, in the see-through region (120) of the head-mounted display device, displaying at least a part of the physical object (130) in a see-through way.

Description

Method, apparatus, device and product for display of head-mounted display device

[0001] This application claims priority to Chinese patent application No. 202410405063.9, filed on April 3, 2024, entitled “Methods, devices, equipment and products for display of head-mounted display devices”. The entire contents of that application are incorporated herein by reference. Technical Field

[0002] The present disclosure generally relates to the field of computers, and more particularly, to methods, apparatuses, electronic devices, and products for displaying a head-mounted display device. Background Art

[0003] Extended Reality (XR) is a hybrid of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. In an XR environment, users can immerse themselves in a completely virtual world through specialized devices, or interact with virtual elements in a real-world setting, experiencing the interplay between virtual and real life. These virtual elements can be any imaginable object, scene, or character. Through high-definition displays, precise positional tracking, and realistic virtual elements, users feel as if they are immersed in a whole new world.

[0004] XR devices are key to achieving this virtual reality experience. These devices typically include components such as a head-mounted display (HMD), interactive controllers, and sensors. The HMD is responsible for presenting images of the virtual world, the interactive controllers allow users to interact with virtual elements, and the sensors track the user's position and movements, ensuring real-time feedback from the virtual world. Summary of the Invention

[0005] Embodiments of the present disclosure provide a method, apparatus, electronic device, and product for displaying a head-mounted display device.

[0006] According to a first aspect of the present disclosure, a method for displaying a head-mounted display device is provided. The method includes obtaining information about a physical object in a physical environment in response to an intersection between a first area of ​​an interactor of the head-mounted display device and a second area of ​​the physical object. The method also includes determining a see-through area for see-through display in the head-mounted display device based on the information about the physical object. The method also includes displaying at least a portion of the physical object in the see-through area of ​​the head-mounted display device.

[0007] In a second aspect of the present disclosure, a display device for a head-mounted display device is provided. The device includes an information acquisition module configured to acquire information about a physical object in a physical environment in response to an intersection between a first area of ​​an interactor of the head-mounted display device and a second area of ​​a physical object. The device also includes a see-through area determination module configured to determine a see-through area for see-through display in the head-mounted display device based on the information about the physical object. Furthermore, the device also includes a display module configured to see-through display at least a portion of the physical object in the see-through area of ​​the head-mounted display device.

[0008] In a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.

[0009] In a fourth aspect of the present disclosure, a computer program product is provided, on which computer executable instructions are stored, wherein the computer executable instructions are executed by a processor to implement the method of the first aspect.

[0010] This summary is intended to introduce a selection of concepts in a simplified form that are further described below in the detailed description. It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] FIG1 illustrates a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;

[0013] FIG2 shows a flowchart of a method for displaying a head-mounted display device according to some embodiments of the present disclosure;

[0014] FIG3A is a schematic diagram showing a display process for a head-mounted display device according to some embodiments of the present disclosure;

[0015] FIG3B shows a schematic diagram of a collision volume of an interactor according to some embodiments of the present disclosure;

[0016] FIG3C is a schematic diagram showing distances between an interactor and grid points on a physical object according to some embodiments of the present disclosure;

[0017] FIG3D shows a schematic diagram of a normalized mapping curve of distances between an interactor and grid points on a physical object according to some embodiments of the present disclosure;

[0018] FIG3E shows a schematic diagram of weights of distances between an interactor and a grid point on a physical object according to some embodiments of the present disclosure;

[0019] FIG3F shows a schematic diagram of the collision center of an interactor according to some embodiments of the present disclosure;

[0020] FIG3G shows a schematic diagram of determining a see-through area according to a collision center of an interactor according to some embodiments of the present disclosure;

[0021] FIG3H shows a schematic diagram for displaying a physical object through some embodiments of the present disclosure;

[0022] FIG4A is a schematic diagram showing another display process for a head-mounted display device according to some embodiments of the present disclosure;

[0023] FIG4B shows a schematic diagram of recording the coordinates of the center point of an interactor at the time of collision after a collision occurs according to some embodiments of the present disclosure;

[0024] FIG4C shows a schematic diagram of some embodiments of the present disclosure for recording the coordinates of the center point of an interactor based on a time interval after a collision occurs;

[0025] FIG4D shows a schematic diagram of some embodiments of the present disclosure for recording the coordinates of the center point of the interactor based on the movement distance of the interactor after a collision occurs;

[0026] FIG4E is a schematic diagram showing a set of recorded interactor center point coordinates according to some embodiments of the present disclosure;

[0027] FIG4F shows a schematic diagram of displaying a physical object according to a degree of trust in some embodiments of the present disclosure;

[0028] FIG5A is a schematic diagram showing another display process for a head-mounted display device according to some embodiments of the present disclosure;

[0029] FIG5B shows a schematic diagram of triggering a see-through area based on the movement speed of a head interactor according to some embodiments of the present disclosure;

[0030] FIG5C is a schematic diagram showing some embodiments of the present disclosure showing triggering of a see-through area based on a distance between a head interactor and a grid point of a physical object reaching a threshold;

[0031] FIG5D is a schematic diagram showing triggering of a see-through area based on the speed of a hand interactor reaching a threshold speed in some embodiments of the present disclosure;

[0032] FIG6 shows a block diagram of an apparatus for displaying a head-mounted display device according to some embodiments of the present disclosure; and

[0033] FIG7 shows a block diagram of an electronic device according to some embodiments of the present disclosure.

[0034] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0035] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0036] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0037] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. can refer to different or the same objects, unless explicitly stated otherwise. Other explicit and implicit definitions may also be included below.

[0038] In MR experiences, relevant technologies establish a safety zone for the user's safety. This zone aims to prevent injuries from excessive movements or accidental collisions during the immersive experience. It also prevents non-users from accidentally entering the experience zone and potentially getting hurt by the user's actions. However, since the safety zone is typically fixed, it can limit the user's range of movement, impacting the user's immersive experience.

[0039] According to the embodiments of the present disclosure, based on the intersection of the interactor and physical objects in the real physical environment, a portion of the area is dynamically and selectively revealed to the user, and a portion of the real physical objects are revealed in the head-mounted display device. This method, which does not require the establishment of a fixed safe area, breaks the limitation of traditional interaction that requires the establishment of a fixed safe area, allowing users to freely interact with the surrounding environment without the constraints of a fixed safe area. This dynamic interaction method not only enhances the user's immersive experience, but also ensures safety during the interaction process, bringing users an unprecedented new interactive experience.

[0040] Figure 1 illustrates a schematic diagram of an example environment 100 in which some embodiments of the present disclosure may be implemented. As shown in Figure 1 , 140 in the example environment 100 is a virtual environment 140 originally presented by a head-mounted display device. In an XR experience, a user can view the virtual environment 140 through a specific head-mounted display device, such as XR glasses, and immerse themselves in a completely virtual environment 140 constructed by a computer system. Alternatively, the user can interact with virtual elements in a real environment, experiencing the interaction between virtual and real life.

[0041] Continuing with FIG1 , when the interactor 110 of the head-mounted display device approaches a certain distance from a real object 130 in the real world, the head-mounted display device will dynamically display an area of ​​a certain size on the user's screen 140, such as the see-through area 120 shown in FIG1 . Within this see-through area 120, the real object 130 that the user is about to encounter will be dynamically displayed. For example, when a sphere representing the interactor 110 with a radius R around its center intersects the real object 130, the camera on the head-mounted display device will capture mesh data of the real object 130 in the physical environment in real time, dynamically determine the collision center of the intersecting real object 130, and then use this collision center to determine the extent of the see-through area 120, which will be dynamically displayed on the user's display device. The size, shape, and effect of the see-through area 120 can be set by the user or automatically configured by a computer system. These interactors can be interactive elements such as handles, hands, or the head.

[0042] The following describes in detail the process according to an embodiment of the present disclosure in conjunction with Figures 2 to 7. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0043] FIG2 shows a flowchart of a method 200 for displaying a head-mounted display device according to some embodiments of the present disclosure. In block 202, in response to an intersection between a first area of ​​an interactor of the head-mounted display device and a second area of ​​a physical object in a physical environment, information of the physical object is obtained. The first area of ​​the interactor of the head-mounted display device refers to an area centered on the center point of the interactor, for example, a spherical area with a radius of R and a center point of the interactor 110. When the spherical area intersects with the physical object 130, the mesh data (polygonal mesh data) of the physical object 130 can be obtained by collecting information of the physical object 130 in the physical environment through the camera on the head-mounted display device.

[0044] At block 204, a see-through area for the see-through display in the head-mounted display device is determined based on the physical object information. After obtaining mesh data for the physical object 130, the see-through display area can be determined based on the obtained mesh data. For example, the center point of the see-through area 120 can be determined using the mesh points of the physical object 130, and the size of the see-through area 120 can be determined based on this center point.

[0045] At block 206, at least a portion of the physical object is displayed see-through in a see-through region of the head mounted display device. After determining the extent of the see-through region 120, at least a portion of the physical object 130 may be displayed in a virtual environment 140 displayed in the head mounted display device.

[0046] In this embodiment, when the outer area of ​​the interactor intersects a real object, information about the object is obtained. Based on this information, a see-through area is determined, within which at least a portion of the real object's content is displayed on the head-mounted display device in a see-through manner. This allows the user to see both the virtual content and the real object integrated with the virtual content, enhancing the visual experience without compromising the user's immersive experience and ensuring safety.

[0047] Figure 3A shows a schematic diagram of a display process 300A for a head-mounted display device in some embodiments of the present disclosure. Referring to Figure 3A, at 310A, a collision event is detected. When the collision body of the interactor intersects with a physical object in the physical environment, it will be determined as a collision event. Then, through a series of calculations and transformations, the potential collision location and the relative degree of distance between the interactor and the collision area can be predicted. At 322A, all the vertices of the Mesh are selected within a spherical area with a radius of R centered on the interactor. Referring to Figure 3B, in 300B, a collision body 320B that can represent the interactor can be obtained. This collision body can be a sphere with a radius of R centered on the interactor 310B. This radius can be set arbitrarily, or it can change according to the movement speed of the interactor. Alternatively, this collision body can also be any other arbitrary shape centered on the center point of the interactor.

[0048] In some embodiments, the interactor can be an interactive element such as a handle, hand, or head, for example, it can also be a ring, bracelet, or other device. In some embodiments, the radius R of the collision body can vary with the movement speed of the interactor. The radius of the interactor in some embodiments can also vary with the degree of danger of the real object. Alternatively, the shape of the collision body 320B can also be other shapes. All the vertices of the Mesh can be selected within this spherical area 320B, as shown in 330C in Figure 3C, and these points are all the vertices of the selected Mesh.

[0049] Continuing with Figure 3A , at 324A, the distance from each Mesh vertex to the sphere center is calculated, and a weight for each vertex is generated. In conjunction with Figure 3C , in 300C, the collision volume is a sphere with a radius R, centered at the interactor center point 320C. This collision volume intersects the real object 310C, and the vertices 330C are selected Mesh vertices within the collision volume area. Next, the distance D 340C between all these selected Mesh vertices and the interactor center point 320C can be calculated. Assume that the mesh has 5 vertices, their coordinates are: V1(1,2,3), V2(4,5,6), V3(7,8,9), V4(10,11,12), V5(13,14,15), and the coordinate of the center of the sphere is C(0,0,0). The distance from each vertex to the center of the sphere can be calculated: D1=3.74, D2=8.66, D3=13.60, D4=18.52, D5=23.44.

[0050] Referring to Figure 3A, for example, when 16 points are selected (not shown in the figure), the distances D 340C, [D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13, D14, D15, D16] between the vertices of the 16 Mesh and the center point 320C of the interactor will be calculated. Then, the weight of each vertex can be obtained for these 16 distances D. Referring to Figure 3D, at 300D, the calculated distance values ​​can be normalized, and then these normalized values ​​can be mapped through an S-shaped curve to generate the weight W of each vertex. In some embodiments, the normalized distances of the 16 points can be counted using a normal distribution, and the mapping process can be performed using the half-edge curve of the normal distribution. In some embodiments, points with normalized distance values ​​close to 1 are selected for subsequent operations. For example, assuming that after 20 points are normalized, 10 of them are within the range of [0.8, 1], then these 10 points can be selected for subsequent operations. Each normalized distance value is mapped to this S-shaped curve to generate the corresponding weight W. The closer the distance value is to 1 (that is, the closer to the center of the sphere), the higher the weight W value after mapping, indicating that the probability of the user touching these vertices is higher. On the contrary, the closer the distance value is to 0, the lower the weight W value, indicating that the probability of the user touching is lower. In conjunction with Figure 3E, in 300E, these points are secondary selected within the range of a sphere with a radius of R and the center of the interactor as the center point to calculate the collision center. As shown in 310E, the weight of each selected vertex is W1, W2, W3, W4, W5, W6, W7, W8, W9, W10.

[0051] Continuing to refer to FIG3A , at 326A, the collision center is determined. This collision center is the center point of the second selected Mesh vertex. In some embodiments, the total weight of the 10 selected Mesh vertices is first calculated, and the total weight can be obtained by adding up the weights W of all vertices. Then the weighted coordinates are calculated, and for each vertex, its X, Y, and Z coordinates are multiplied by the corresponding weight W, so that the weighted coordinates can be obtained. Then the weighted coordinates are divided by the total weight to obtain the coordinates of the center point O. For example, the total weight can be calculated as W1+W2+W3+W4+W5+W6+W7+W8+W9+…+Wn. Then the weighted coordinates are calculated, and the coordinates of each vertex are multiplied by their respective weights to obtain the weighted coordinates. Weighted x-coordinate = W1x1 + W2x2 + W3x3 + ... + Wnxn; weighted y-coordinate = W1y1 + W2y2 + W3y3 + ... + Wnyn; weighted z-coordinate = W1z1 + W2z2 + W3z3 + ... + Wnzn. Next, determine the center point O of the collision probability by dividing the weighted coordinates by the total weight to obtain the coordinates of center point O. O's x-coordinate = (weighted x-coordinate) / total weight; O's y-coordinate = (weighted y-coordinate) / total weight; O's z-coordinate = (weighted z-coordinate) / total weight. This determines the coordinates of collision center 310F in Figure 3F.

[0052] As shown in FIG3F , in some embodiments, the distance 320F can be determined using the determined collision center 310F and the center point of the interactor, as well as the radius R of the interactor's collision volume. This facilitates subsequent calculation of the size of the collision area.

[0053] Continuing to refer to FIG3A , at 328A, the see-through area is calculated. In some embodiments, after the collision center O is determined, the distance between the collision center O and the center of the interactor can be calculated. As shown in FIG3F , at 300F, the distance L between the collision center 310F and the center of the interactor is 320F. In some embodiments, the range of the see-through area can be determined based on the relationship between the distance L between the collision center and the center of the interactor and the radius R of the collision body of the interactor. For example, L can be divided by R to obtain a metric I (normalized) close to the collision range, and then the value of I can be mapped through an S-shaped curve using normal distribution statistics as a numerical reference for optimizing the range of the see-through part. The curve mapping of the metric I is related to the visual effect of the output. The higher the probability of touch, the greater the intensity / range of the see-through part. In some embodiments, the higher the probability of being touched by the user, the greater the intensity and range of the see-through part. In some embodiments, a sphere can be drawn at the collision center to show the real-world image within the spherical range. As shown in FIG. 3G , at 300G, the transmission area 320G is a spherical area with a radius of 330G and a collision center 310G as the center.

[0054] Continuing with FIG3A , at 330A, visualization output is performed. Once the transparency area is determined, the real-world image can be projected within the transparency area. As shown in FIG3H , at 300H, the real image 320H is a spherical area with a radius of I and a collision center 310H as the center. In some embodiments, the shape of the transparency area is not fixed and can be other shapes with the collision center as the center point. In some embodiments, the blur radius, radius, and transparency of the transparency area can all be set. It is understandable that these visualization effects are not unique.

[0055] Based on the collision event, the distance detection between the mesh vertices of the environment space and the interactor is triggered, and the center of the transmittance area and the transmittance intensity at each location are determined using the mesh vertex position. This method significantly improves the accuracy of the interactor's collision warning function and ensures the safety and experience quality of users during the interaction process.

[0056] Figure 4A shows a schematic diagram of another display process 400A for a head-mounted display device according to some embodiments of the present disclosure. As shown in Figure 4A, at 410A, a specific area is set with the interactor as the center. Referring to Figure 3B, this specific area is a sphere 320B with the center of the watering can 310B as the center and R as the radius. At 420A, the coordinates of the interactor in the air are recorded. In some embodiments, when the specific area centered on the interactor intersects with an object, the coordinates of the center point of the interactor in the air are recorded. For example, each time the specific area centered on the interactor collides with an object, the center point of the interactor at each collision is recorded at 422A. As shown in Figure 4B, at 400B, the interactor moves toward the object 420B. After the specific area centered on the interactor collides with the object 420B, the center point of the interactor at the time of the collision is recorded. For example, after a specific area 412B centered on the interactor collides with object 420B, the coordinates of the center point of the interactor at that time may be recorded. For another example, after a specific area 414B centered on the interactor collides with 420B again, the coordinates of the center point of the interactor at that time may be recorded. For another example, after a specific area 416B centered on the interactor collides with 420B again, the coordinates of the center point of the interactor at that time may be recorded.

[0057] For another example, when the range of a specific area centered on the interactor collides with an object, at 424A, regardless of whether the interactor moves, the coordinates of the center point of the interactor will be recorded at intervals. As shown in Figure 4C, at 400C, when the range 410C of the specific area centered on the interactor collides with the object, the coordinates of the center point of the interactor at that time can be recorded. In some embodiments, after the collision occurs, regardless of whether the interactor moves, the center coordinate position of the interactor will be recorded at intervals. For example, after a period of time, the range of the specific area centered on the interactor is moved to 412C, and the coordinate position of the center point of the interactor at that time can be recorded. For another example, after a period of time, the coordinate position of the center point of the interactor in the range 414C of the specific area centered on the interactor will be recorded. For another example, after another period of time, the coordinate position of the center point of the interactor in the range 416C of the specific area centered on the interactor will be recorded.

[0058] For another example, when a specific area centered on the interactor collides with an object, at 426A, the coordinates of the center point of the interactor are recorded each time the interactor moves a certain distance. As shown in FIG4D , at 400D, when a specific area 410D centered on the interactor collides with an object, the coordinates of the center point of the interactor at that time can be recorded. In some embodiments, after a collision occurs, when the center position of the interactor is removed from the area with a radius R set in the previous frame (i.e., the distance moved is greater than the radius R), the center position of the current specific area 412D centered on the interactor is recorded.

[0059] For another example, in 428A, when a specific area centered on the interactor collides with an object, and the interactor's movement speed exceeds a certain value, the center coordinates of the interactor may be recorded at regular intervals. For example, when the interactor's movement speed exceeds 1 meter per second, the center coordinates of the interactor may be recorded twice per second (2 times / second).

[0060] Continuing with reference to FIG4A , at 430A, the number of points within a certain radius of the interactor and the probability distribution of trust are counted. In some embodiments, each time a collision occurs between a specific area centered on the interactor and an object, the number of points within a certain radius of the interactor can be counted, for example, the number of points within a spherical area with a radius of R and a sphere centered on the interactor is counted. In some embodiments, the distance D from each point in these areas to the center of the interactor is calculated, and then D is divided by R to obtain a normalized distance. Next, the normalized distance is mapped to an S-shaped curve through a normal distribution, and used as the weight W of each recorded center coordinate position. The probability of the trust of the vertex with a value close to 1 will be higher.

[0061] As shown in Figure 4E, once all points have been recorded, a spherical region can be constructed with the center of the interactor at that moment as a sphere and a radius of R. The trustworthiness of the recorded points within this spherical region is then calculated. In some embodiments, the distance between the recorded points within this spherical region and the center of the interactor at that moment can be calculated, and the weights of the recorded points within this region can be calculated through normalization. The trustworthiness of these points within this region is then determined based on these weights.

[0062] As shown in Figure 4E , points 400E and 410E are considered highly trusted, 420E is considered the second most trusted, 430E is considered the least trusted, and 440E is considered the least trusted. It's understandable that the trustworthiness of these points isn't constant. Over time and as the user operates the interactor, the coordinates of new interactor center points may be recorded. Simultaneously, the trustworthiness of these points is recalculated.

[0063] Continuing with FIG4A , at 440A, a determination is made as to whether a repeated warning should be issued for the collision region based on the settled confidence level. In some embodiments, when the confidence level of the collision region is high, no warning may be issued to the user, and the real world may not be revealed to the user. When the confidence level of the collision region is low, the user is warned, and the real world is revealed to the user. Continuing with FIG4A , at 450A, as the confidence level increases, the warning effect becomes less pronounced. As the confidence level decreases, the warning effect becomes more pronounced, such as by increasing transparency, blurring, and radius.

[0064] Referring to FIG. 4F , at 400F, as shown in 410F, when the trust level is the highest, the real world will not be visible. As shown in 420F, when the trust level is the second highest, the real world will be slightly visible. As shown in 430F, when the trust level is low, the real world will be slightly visible. As shown in 440F, when the trust level is the lowest, the real world will be more clearly visible. In some embodiments, the size of these visible areas and the effect of the visible area will vary with the trust level. The lower the trust level, the more obvious the size and effect of the visible area, and vice versa.

[0065] By dynamically counting the coordinates of the interactors of the user's actions in areas that have been marked as dangerous, it is possible to dynamically determine areas where the user has high interaction needs, mark these areas with interaction needs as high trust, and no longer issue warnings, thereby improving the user's immersive experience.

[0066] Figure 5A illustrates another schematic diagram of a display process 500A for a head-mounted display device according to some embodiments of the present disclosure. Referring to Figure 5A , at 510A, based on different interactor collision behavior rules, the control area is partially exposed, providing a user with a warning. In some embodiments, the interactor collision behavior rules are categorized into two main types: hand and head. Hand types include the hand itself, as well as various hand-related interactors, such as handles, bracelets, media, and other wearable devices.

[0067] Continuing with FIG. 5A , at 522A, when the head interactor's movement speed reaches a certain threshold, a reveal prompt is issued. For example, when the head movement speed reaches 1 m / s, the current real world is revealed within the user's field of view. As shown in FIG. 5B , at 500B, when the head interactor 520B reaches a certain movement speed, the current real world is revealed within the field of view 510B of the head interactor 520B. For example, when the head movement speed reaches 1 m / s, the current real world is revealed within the user's field of view.

[0068] In some embodiments, the size and position of the see-through area can be determined based on the size of the head interactor's collision volume, the distance between the head interactor's center point and the collision center, and the head interactor's speed. For example, the greater the head interactor's speed, the larger the head interactor's collision volume may be, resulting in more vertices in the intersection area with the real world. This will affect the determination of the collision center, and thus the distance between the collision center and the interactor's center point, ultimately determining the size of the see-through area.

[0069] Continuing with Figure 5A , at 524A, a see-through prompt is triggered when the head collision volume collides with the real world or the distance between the head, such as the face, and the real world reaches a predetermined threshold. In some embodiments, a see-through prompt can be issued to the user when the outer boundary of the head collision volume is less than 0.1 meters or less than 0.3 meters from the real world. As shown in Figure 5C , at 500C, when the collision volume 510C of the head interactor reaches a danger distance threshold 520C, a see-through prompt is issued to the user. In some embodiments, a see-through prompt is issued when the head collision volume intersects the real world during movement. The see-through area can be determined based on the location where the collision volume intersects the real world and the speed of the interactor. The faster the interactor, the larger the see-through area. The larger the collision volume, the larger the see-through area. In some embodiments, if the system identifies sharp objects or high-risk physical properties in the real world, the see-through area of ​​the collision volume will be larger and the intensity of the see-through will be greater, thereby increasing the user's vigilance. Conversely, when the danger level is lower, the see-through area can be set to be smaller.

[0070] Continuing with FIG5A , at 532A, the required buffer distance for the hand interactor's current real-time velocity is calculated. As shown in FIG5D , at 500D, the required buffer distance 526D for a collision between the interactor 520D and the obstacle 510D can be predicted based on the hand interactor's velocity. In some embodiments, when the velocity of the interactor 520D is known, the required buffer distance 526D before contact with the obstacle 510D can be calculated in advance.

[0071] Continuing to refer to Figure 5A, at 534A, the offset of the interactor collision body is controlled. In some embodiments, the offset 524D of the interactor's buffer collision body 522D based on the interactor center can be calculated based on the buffer distance 526D. Then, the position and size of the buffer collision body are adjusted accordingly. In some embodiments, the greater the movement speed of the hand interactor, the greater the buffer distance of the buffer collision body, and the greater the offset. By adjusting the size of the buffer collision body according to the movement speed of the hand interactor, it is possible to warn or display dangerous areas in advance, thereby giving the user enough time to avoid or change direction.

[0072] In some embodiments, the see-through area can be implemented in a variety of ways. For example, adjustments to blur, radius, transparency, edge softening, or the addition of warning icons can be used. This approach effectively alerts users to potential dangers without overly disrupting them. Furthermore, this method can adjust the size and position of the see-through area in real time as the user's actions and interaction state change, adapting to changing scenarios and environments.

[0073] Figure 6 shows a block diagram of an apparatus 600 for displaying a head-mounted display device according to some embodiments of the present disclosure. As shown in Figure 6, the apparatus 600 includes an information acquisition module 602, which is configured to acquire information about a physical object in response to an intersection between a first area of ​​an interactor of the head-mounted display device and a second area of ​​a physical object in a physical environment. The apparatus 600 also includes a see-through area determination module 604, which is configured to determine a see-through area to be displayed see-through in the head-mounted display device based on the information about the physical object. In addition, the apparatus 600 also includes a display module 606, which is configured to display at least a portion of the physical object see-through in the see-through area of ​​the head-mounted display device.

[0074] FIG7 shows a block diagram of an electronic device 700 of some embodiments of the present disclosure, and device 700 may be the device or apparatus described in an embodiment of the present disclosure. As shown in FIG7 , device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. In RAM 703, various programs and data required for the operation of device 700 can also be stored. CPU / GPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704. Although not shown in FIG7 , device 700 may also include a coprocessor.

[0075] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0076] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0077] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0078] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0079] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0080] The computer program instructions for performing the disclosed operation can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or the object code written in any combination of one or more programming languages, programming languages ​​include object-oriented programming languages, and conventional procedural programming languages.Computer-readable program instructions can be performed completely on a user's computer, partially on a user's computer, performed as an independent software package, partly on a user's computer and partly on a remote computer, or performed completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network-including local area network (LAN) or wide area network (WAN), or can be connected to an external computer (such as utilizing an internet service provider to connect by the internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to carry out personalized customization electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLA), this electronic circuit can perform computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0081] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0083] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented by a special hardware-based system that performs the prescribed function or action, or can be implemented by a combination of special hardware and computer instructions.

[0084] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements to existing technologies, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

[0085] Some example implementations of the present disclosure are listed below.

[0086] Example 1. A method for displaying a head-mounted display device, comprising:

[0087] In response to a first area of ​​the interactor of the head mounted display device intersecting a second area of ​​a physical object in a physical environment, obtaining information of the physical object;

[0088] determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object; and

[0089] In the see-through area of ​​the head-mounted display device, at least a portion of the physical object is displayed see-through.

[0090] Example 2. The method of Example 1, wherein in response to a first area of ​​the interactor of the head-mounted display device intersecting a second area of ​​a physical object in the physical environment, obtaining information about the physical object comprises:

[0091] determining a plurality of grid points based on the first area of ​​the interactor; and

[0092] A collision center of the physics object is determined based on the plurality of grid points.

[0093] Example 3. The method of any of Examples 1-2, wherein determining a collision center of the physics object based on the plurality of grid points comprises:

[0094] determining a plurality of distances between a plurality of the grid points and a center point of the interactor;

[0095] determining a plurality of weights for the plurality of grid points based on the plurality of distances; and

[0096] The collision center is determined based on the plurality of weights of the plurality of grid points.

[0097] Example 4. The method of any one of Examples 1-3, wherein determining a plurality of weights for the plurality of grid points based on the plurality of distances comprises:

[0098] determining a plurality of normalized values ​​for the plurality of distances based on the plurality of distances; and

[0099] The plurality of weights of a plurality of the grid points are determined based on the plurality of normalized values.

[0100] Example 5. The method of any of Examples 1-4, wherein determining the collision center based on the plurality of weights of the plurality of grid points comprises:

[0101] determining a plurality of calculation points from the plurality of grid points based on the plurality of weights of the plurality of grid points; and

[0102] The collision center is determined based on a plurality of weights of the plurality of calculation points.

[0103] Example 6. The method of any of Examples 1-5, wherein determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object comprises:

[0104] The see-through area is determined based on a distance between the center point of the interactor and the collision center and a length from the center point of the interactor to an outer boundary of a first area of ​​the interactor.

[0105] Example 7. The method of any of Examples 1-6, wherein, in the see-through area of ​​the head-mounted display device, see-through displaying at least a portion of the physical object comprises:

[0106] Based on the transparency area, at least a portion of the physical object is transparently displayed in a feathered manner.

[0107] Example 8. The method of any of Examples 1-7, wherein determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object comprises:

[0108] In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, determining a set of center point coordinates of the interactor; and

[0109] Based on the center point coordinate set, multiple confidence levels of multiple center points are determined.

[0110] Example 9. The method of any of Examples 1-8, wherein determining a set of center point coordinates of the interactor comprises at least one of the following:

[0111] In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, recording the coordinates of the center point of the interactor;

[0112] In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, recording the coordinates of the center point of the interactor based on a certain time interval;

[0113] In response to the first area of ​​the interactor of the head-mounted display device intersecting the second area of ​​the physical object in the physical environment, and in response to the distance between the center point of the interactor at a current moment and the center point of the interactor at a previous moment being greater than the length from the center point of the interactor at the current moment to the outer boundary of the first area, recording the coordinates of the center point of the interactor at the current moment; or

[0114] In response to the movement speed of the interactor reaching a first speed threshold, the coordinates of the center point of the interactor are recorded based on a certain time interval.

[0115] Example 10. The method of any one of Examples 1-9, wherein determining a plurality of trust levels of a plurality of the center points based on the set of center point coordinates comprises:

[0116] In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, determining a subset of the set of center point coordinates of the interactor based on a length from the center point of the interactor to an outer boundary of the first area and the center point of the interactor;

[0117] Based on the subset, determining distances between a plurality of points in the subset and a center point of the interactor;

[0118] determining a plurality of weights for a plurality of points in the subset based on the distance; and

[0119] A confidence level of each of the plurality of points in the subset is determined based on the plurality of weights of the plurality of points.

[0120] Example 11. The method of any of Examples 1-10, wherein determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object comprises:

[0121] In response to the confidence level being within a first range, at least a portion of the physical object is displayed through; and in response to the confidence level being within a second range, the display of the physical object through the display is stopped.

[0122] Example 12. The method of any of Examples 1-11, wherein determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object comprises:

[0123] determining whether the interactor is located on the hand or the head;

[0124] In response to the interactive device being located on the hand, determining a see-through area for the hand; and

[0125] In response to the interactor being located at the head, a see-through area for the head is determined.

[0126] Example 13. The method of any one of Examples 1-12, wherein, in response to the interactive device being worn on a hand, determining a see-through area for the hand comprises:

[0127] In response to the interactive device being located on the hand, determining a buffer distance based on a movement speed of the interactive device;

[0128] determining an offset of the first area of ​​the interactor based on the buffer distance;

[0129] determining a size of the first area of ​​the interactor based on the offset; and

[0130] The see-through area for the hand is determined based on the size of the first area, the distance between the center point of the interactor and the collision center, and the length from the center point of the interactor to the outer boundary of the first area of ​​the interactor.

[0131] Example 14. The method of any one of Examples 1-13, wherein, in response to the interactive device being worn on a head, determining a see-through area for the head comprises:

[0132] In response to the interactive device being worn on the head and in response to the speed of the interactive device satisfying a second speed threshold, the see-through area for the head is determined within the field of view of the interactive device.

[0133] Example 15. The method of any one of Examples 1-14, further comprising:

[0134] In response to the interactor being worn on the head and in response to the distance between the outer boundary of the first area of ​​the interactor and the physical environment meeting a distance threshold, the see-through area for the head is determined within the field of view of the interactor.

[0135] Example 16. The method of any of Examples 1-15, wherein determining the show-through area for the head comprises:

[0136] The see-through area for the head is determined based on a position where the first area of ​​the interactor intersects a second area of ​​the physical object in the physical environment and a speed of the interactor.

[0137] Example 17. An apparatus for displaying a head-mounted display device, comprising:

[0138] an information acquisition module configured to acquire information of a physical object in a physical environment in response to an intersection between a first area of ​​an interactor of the head mounted display device and a second area of ​​the physical object;

[0139] a see-through area determination module configured to determine a see-through area for see-through display in the head mounted display device based on information about the physical object; and

[0140] The display module is configured to transparently display at least a portion of the physical object in the transparent area of ​​the head-mounted display device.

[0141] Example 18. The apparatus of Example 17, wherein the information acquisition module comprises:

[0142] a grid point determination module configured to determine a plurality of grid points based on the first area of ​​the interactor; and

[0143] The collision center determination module is configured to determine the collision center of the physical object based on the multiple grid points.

[0144] Example 19. The apparatus of any of Examples 17-18, wherein the collision center determination module comprises:

[0145] a distance determination module configured to determine a plurality of distances between a plurality of the grid points and a center point of the interactor;

[0146] A first weight determination module is configured to determine a plurality of weights of the plurality of grid points based on the plurality of distances; and

[0147] The first collision center determination module is configured to determine the collision center based on the multiple weights of the multiple grid points.

[0148] Example 20. The apparatus of any of Examples 17-19, wherein the weight determination module comprises:

[0149] a normalization value determination module configured to determine a plurality of normalization values ​​of the plurality of distances based on the plurality of distances; and

[0150] The second weight determination module is configured to determine the multiple weights of the multiple grid points based on the multiple normalized values.

[0151] Example 21. The apparatus of any one of Examples 17-20, wherein the first collision center determination module comprises:

[0152] a calculation point determination module configured to determine a plurality of calculation points from the plurality of grid points based on the plurality of weights of the plurality of grid points; and

[0153] The second collision center determination module is configured to determine the collision center based on a plurality of weights of the plurality of calculation points.

[0154] Example 22. The apparatus of any one of Examples 17-21, wherein the transmittance region determination module comprises:

[0155] The first see-through area determination module is configured to determine the see-through area based on the distance between the center point of the interactor and the collision center and the length from the center point of the interactor to the outer boundary of the first area of ​​the interactor.

[0156] Example 23. The apparatus of any of Examples 17-22, wherein the display module comprises:

[0157] The first display module is configured to transparently display at least a portion of the physical object in a feathered manner based on the transparent area.

[0158] Example 24. The apparatus of any of Examples 17-23, wherein the transmittance region determination comprises:

[0159] a center point coordinate set determining module configured to determine a center point coordinate set of the interactor in response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment; and

[0160] The trust degree determination module is configured to determine multiple trust degrees of multiple center points based on the center point coordinate set.

[0161] Example 25. The apparatus of any of Examples 17-24, wherein the module for determining a set of center point coordinates comprises at least one of the following:

[0162] a first center point coordinate recording module configured to record the center point coordinates of the interactor in response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment;

[0163] a second center point coordinate recording module, configured to record the center point coordinates of the interactor based on a certain time interval in response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment;

[0164] a third center point coordinate recording module, configured to, in response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, and in response to the distance between the center point of the interactor at a current moment and the center point of the interactor at a previous moment being greater than the length from the center point of the interactor at the current moment to the outer boundary of the first area, record the center point coordinates of the interactor at the current moment; or

[0165] The fourth center point coordinate recording module is configured to record the center point coordinates of the interactor based on a certain time interval in response to the moving speed of the interactor reaching a first speed threshold.

[0166] Example 26. The apparatus of any of Examples 17-25, wherein the trustworthiness determination module comprises:

[0167] a subset determination module configured to, in response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, determine a subset of the center point coordinate set of the interactor based on a length from the center point of the interactor to an outer boundary of the first area and the center point of the interactor;

[0168] a first distance determination module configured to determine, based on the subset, distances between a plurality of points in the subset and a center point of the interactor;

[0169] a third weight determination module configured to determine a plurality of weights of a plurality of points in the subset based on the distance; and

[0170] The first confidence determination module is configured to determine the confidence of each point in the plurality of points in the subset based on the plurality of weights of the plurality of points.

[0171] Example 27. The apparatus of any one of Examples 17-26, wherein the transmittal area display module comprises:

[0172] a second display module configured to transparently display at least a portion of the physical object in response to the trust level being within a first range; and

[0173] The display stopping module is configured to stop transparently displaying the physical object in response to the trust level being in a second range.

[0174] Example 28. The apparatus of any of Examples 17-27, wherein the transmittance region determination module comprises:

[0175] a determination module configured to determine whether the interactor is located on the hand or the head;

[0176] a second see-through area determination module configured to determine a see-through area for the hand in response to the interactor being located on the hand; and

[0177] The third see-through area determination module is configured to determine a see-through area for the head in response to the interactor being located at the head.

[0178] Example 29. The apparatus of any one of Examples 17-28, wherein the second transmittance area determination module comprises:

[0179] a buffer distance determination module configured to determine a buffer distance based on a movement speed of the interactor in response to the interactor being located on the hand;

[0180] an offset determination module configured to determine an offset of the first area of ​​the interactor based on the buffer distance;

[0181] a size determination module configured to determine a size of the first area of ​​the interactor based on the offset; and

[0182] The fourth see-through area determination module is configured to determine the see-through area for the hand based on the size of the first area, the distance between the center point of the interactor and the collision center, and the length from the center point of the interactor to the outer boundary of the first area of ​​the interactor.

[0183] Example 30. The apparatus of any one of Examples 17-29, wherein the third transmittance area determination module comprises:

[0184] The fifth see-through area determination module is configured to determine the see-through area for the head within the field of view of the interactor in response to the interactor being worn on the head and in response to the speed of the interactor meeting a second speed threshold.

[0185] Example 31. The apparatus of any of Examples 17-30, further comprising:

[0186] The sixth see-through area determination module is configured to determine the see-through area for the head within the field of view of the interactor in response to the interactor being worn on the head and in response to the distance between the outer boundary of the first area of ​​the interactor and the physical environment meeting a distance threshold.

[0187] Example 32. The apparatus of any of Examples 17-31, wherein determining the transmittance region determination module comprises:

[0188] A sixth see-through area determination module is configured to determine the see-through area for the head based on a position where the first area of ​​the interactor intersects with a second area of ​​the physical object in the physical environment and a speed of the interactor.

[0189] Example 33. An electronic device comprising:

[0190] processor; and

[0191] A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs actions, the actions comprising:

[0192] In response to a first area of ​​the interactor of the head mounted display device intersecting a second area of ​​a physical object in a physical environment, obtaining information of the physical object;

[0193] determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object; and

[0194] In the see-through area of ​​the head-mounted display device, at least a portion of the physical object is displayed see-through.

[0195] Example 34. The electronic device of Example 33, wherein in response to a first area of ​​the interactor of the head mounted display device intersecting a second area of ​​a physical object in a physical environment, obtaining information about the physical object comprises:

[0196] determining a plurality of grid points based on the first area of ​​the interactor; and

[0197] A collision center of the physics object is determined based on the plurality of grid points.

[0198] Example 35. The electronic device of any of Examples 33-34, wherein determining a collision center of the physical object based on the plurality of grid points comprises:

[0199] determining a plurality of distances between a plurality of the grid points and a center point of the interactor;

[0200] determining a plurality of weights for the plurality of grid points based on the plurality of distances; and

[0201] The collision center is determined based on the plurality of weights of the plurality of grid points.

[0202] Example 36. The electronic device of any of Examples 33-35, wherein determining a plurality of weights for the plurality of grid points based on the plurality of distances comprises:

[0203] determining a plurality of normalized values ​​for the plurality of distances based on the plurality of distances; and

[0204] The plurality of weights of a plurality of the grid points are determined based on the plurality of normalized values.

[0205] Example 37. The electronic device of any of Examples 33-36, wherein determining the collision center based on the plurality of weights of the plurality of grid points comprises:

[0206] determining a plurality of calculation points from the plurality of grid points based on the plurality of weights of the plurality of grid points; and

[0207] The collision center is determined based on a plurality of weights of the plurality of calculation points.

[0208] Example 38. The electronic device of any of Examples 33-37, wherein determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object comprises:

[0209] The see-through area is determined based on a distance between the center point of the interactor and the collision center and a length from the center point of the interactor to an outer boundary of a first area of ​​the interactor.

[0210] Example 39. The electronic device of any of Examples 33-38, wherein, in the see-through area of ​​the head-mounted display device, the see-through display of at least a portion of the physical object comprises:

[0211] Based on the transparency area, at least a portion of the physical object is transparently displayed in a feathered manner.

[0212] Example 40. The electronic device of any of Examples 33-39, wherein determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object comprises:

[0213] In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, determining a set of center point coordinates of the interactor; and

[0214] Based on the center point coordinate set, multiple confidence levels of multiple center points are determined.

[0215] Example 41. The electronic device of any of Examples 33-40, wherein determining a set of center point coordinates of the interactor comprises at least one of:

[0216] In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, recording the coordinates of the center point of the interactor;

[0217] In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, recording the coordinates of the center point of the interactor based on a certain time interval;

[0218] In response to the first area of ​​the interactor of the head-mounted display device intersecting the second area of ​​the physical object in the physical environment, and in response to the distance between the center point of the interactor at a current moment and the center point of the interactor at a previous moment being greater than the length from the center point of the interactor at the current moment to the outer boundary of the first area, recording the coordinates of the center point of the interactor at the current moment; or

[0219] In response to the movement speed of the interactor reaching a first speed threshold, the coordinates of the center point of the interactor are recorded based on a certain time interval.

[0220] Example 42. The electronic device of any of Examples 33-41, wherein determining a plurality of trust levels of a plurality of the center points based on the set of center point coordinates comprises:

[0221] In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, determining a subset of the set of center point coordinates of the interactor based on a length from the center point of the interactor to an outer boundary of the first area and the center point of the interactor;

[0222] Based on the subset, determining distances between a plurality of points in the subset and a center point of the interactor;

[0223] determining a plurality of weights for a plurality of points in the subset based on the distance; and

[0224] A confidence level of each of the plurality of points in the subset is determined based on the plurality of weights of the plurality of points.

[0225] Example 43. The electronic device of any of Examples 33-42, wherein determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object comprises:

[0226] In response to the confidence level being within a first range, at least a portion of the physical object is displayed through; and in response to the confidence level being within a second range, the display of the physical object through the display is stopped.

[0227] Example 44. The electronic device of any of Examples 33-43, wherein determining, based on the information of the physical object, a see-through area for see-through display in the head-mounted display device comprises:

[0228] determining whether the interactor is located on the hand or the head;

[0229] In response to the interactive device being located on the hand, determining a see-through area for the hand; and

[0230] In response to the interactor being located at the head, a see-through area for the head is determined.

[0231] Example 45. The electronic device of any of Examples 33-44, wherein, in response to the interactive device being worn on a hand, determining a see-through area for the hand comprises:

[0232] In response to the interactive device being located on the hand, determining a buffer distance based on a movement speed of the interactive device;

[0233] determining an offset of the first area of ​​the interactor based on the buffer distance;

[0234] determining a size of the first area of ​​the interactor based on the offset; and

[0235] The see-through area for the hand is determined based on the size of the first area, the distance between the center point of the interactor and the collision center, and the length from the center point of the interactor to the outer boundary of the first area of ​​the interactor.

[0236] Example 46. The electronic device of any of Examples 33-45, wherein, in response to the interactive device being worn on a head, determining a see-through area for the head comprises:

[0237] In response to the interactive device being worn on the head and in response to the speed of the interactive device satisfying a second speed threshold, the see-through area for the head is determined within the field of view of the interactive device.

[0238] Example 47. The electronic device of any of Examples 33-46, further comprising:

[0239] In response to the interactor being worn on the head and in response to the distance between the outer boundary of the first area of ​​the interactor and the physical environment meeting a distance threshold, the see-through area for the head is determined within the field of view of the interactor.

[0240] Example 48. The electronic device of any of Examples 33-47, wherein determining the see-through area for the head comprises:

[0241] The see-through area for the head is determined based on a position where the first area of ​​the interactor intersects a second area of ​​the physical object in the physical environment and a speed of the interactor.

[0242] Example 49. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of Examples 1 to 16.

[0243] Example 50. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method of any one of Examples 1 to 16.

[0244] Although the present disclosure has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A display method for a head-mounted display device, comprising: In response to a first area of ​​the interactor of the head mounted display device intersecting a second area of ​​a physical object in a physical environment, obtaining information of the physical object; determining a see-through area for see-through display in the head-mounted display device based on the information of the physical object; as well as In the see-through area of ​​the head-mounted display device, at least a portion of the physical object is displayed see-through.

2. The method of claim 1 , wherein in response to a first area of ​​the interactor of the head mounted display device intersecting a second area of ​​a physical object in the physical environment, obtaining information about the physical object comprises: determining a plurality of grid points based on the first area of ​​the interactor; as well as A collision center of the physics object is determined based on the plurality of grid points.

3. The method of claim 2 , wherein determining the collision center of the physical object based on the plurality of grid points comprises: determining a plurality of distances between a plurality of the grid points and a center point of the interactor; determining a plurality of weights for the plurality of grid points based on the plurality of distances; as well as The collision center is determined based on the plurality of weights of the plurality of grid points.

4. The method according to claim 3, wherein determining a plurality of weights of the plurality of grid points based on the plurality of distances comprises: determining a plurality of normalized values ​​of the plurality of distances based on the plurality of distances; as well as The plurality of weights of a plurality of the grid points are determined based on the plurality of normalized values.

5. The method of claim 3 , wherein determining the collision center based on the plurality of weights of the plurality of grid points comprises: determining a plurality of calculation points from the plurality of grid points based on the plurality of weights of the plurality of grid points; as well as The collision center is determined based on a plurality of weights of the plurality of calculation points.

6. The method according to claim 3, wherein determining a see-through area for see-through display in the head mounted display device based on the information of the physical object comprises: The see-through area is determined based on a distance between the center point of the interactor and the collision center and a length from the center point of the interactor to an outer boundary of a first area of ​​the interactor.

7. The method of claim 6 , wherein displaying at least a portion of the physical object in the see-through area of ​​the head mounted display device comprises: Based on the transparency area, at least a portion of the physical object is transparently displayed in a feathered manner.

8. The method according to claim 1 , wherein determining a see-through area for see-through display in the head mounted display device based on the information of the physical object comprises: In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, determining a set of center point coordinates of the interactor; as well as Based on the center point coordinate set, multiple confidence levels of multiple center points are determined.

9. The method according to claim 8, wherein determining the center point coordinate set of the interactor comprises at least one of the following: In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, recording the coordinates of the center point of the interactor; In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, recording the coordinates of the center point of the interactor based on a certain time interval; In response to the first area of ​​the interactor of the head-mounted display device intersecting the second area of ​​the physical object in the physical environment, and in response to the distance between the center point of the interactor at a current moment and the center point of the interactor at a previous moment being greater than the length from the center point of the interactor at the current moment to the outer boundary of the first area, recording the coordinates of the center point of the interactor at the current moment; or In response to the movement speed of the interactor reaching a first speed threshold, the coordinates of the center point of the interactor are recorded based on a certain time interval.

10. The method according to claim 8, wherein determining a plurality of confidence levels of a plurality of the center points based on the center point coordinate set comprises: In response to the first area of ​​the interactor of the head mounted display device intersecting the second area of ​​the physical object in the physical environment, determining a subset of the set of center point coordinates of the interactor based on a length from a center point of the interactor to an outer boundary of the first area and the center point of the interactor; Based on the subset, determining distances between a plurality of points in the subset and a center point of the interactor; determining a plurality of weights for a plurality of points in the subset based on the distance; as well as A confidence level of each of the plurality of points in the subset is determined based on the plurality of weights of the plurality of points.

11. The method according to claim 8, wherein determining a see-through area for see-through display in the head mounted display device based on the information of the physical object comprises: In response to the trust level being within a first range, displaying at least a portion of the physical object through-the-lens; as well as In response to the trust level being within a second range, stopping the see-through display of the physical object.

12. The method according to claim 2, wherein determining a see-through area for see-through display in the head mounted display device based on the information of the physical object comprises: determining whether the interactor is located on the hand or the head; In response to the interactive device being located on the hand, determining a see-through area for the hand; as well as In response to the interactor being located at the head, a see-through area for the head is determined.

13. The method according to claim 12, wherein in response to the interactive device being worn on a hand, determining the see-through area for the hand comprises: In response to the interactive device being located on the hand, determining a buffer distance based on a movement speed of the interactive device; determining an offset of the first area of ​​the interactor based on the buffer distance; determining a size of the first area of ​​the interactor based on the offset; as well as The see-through area for the hand is determined based on the size of the first area, the distance between the center point of the interactor and the collision center, and the length from the center point of the interactor to the outer boundary of the first area of ​​the interactor.

14. The method according to claim 12, wherein in response to the interactive device being worn on a head, determining a see-through area for the head comprises: In response to the interactive device being worn on the head and in response to the speed of the interactive device satisfying a second speed threshold, the see-through area for the head is determined within the field of view of the interactive device.

15. The method according to claim 14, further comprising: In response to the interactor being worn on the head and in response to the distance between the outer boundary of the first area of ​​the interactor and the physical environment meeting a distance threshold, the see-through area for the head is determined within the field of view of the interactor.

16. The method according to any one of claims 14-15, wherein determining the show-through area for the head comprises: The see-through area for the head is determined based on a position where the first area of ​​the interactor intersects a second area of ​​the physical object in the physical environment and a speed of the interactor.

17. An apparatus for displaying a head-mounted display device, comprising: an information acquisition module configured to acquire information of a physical object in a physical environment in response to an intersection between a first area of ​​an interactor of the head mounted display device and a second area of ​​the physical object; a see-through area determination module, configured to determine a see-through area for see-through display in the head-mounted display device based on information about the physical object; as well as The display module is configured to transparently display at least a portion of the physical object in the transparent area of ​​the head-mounted display device.

18. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs the method according to any one of claims 1 to 16.

19. A computer program product comprising computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the method according to any one of claims 1 to 16.

Images