Interaction method and electronic device

Abstract

The application discloses an interaction method and an electronic device. It belongs to the technical field of extended reality. An embodiment of the method comprises: acquiring environment data of a physical environment in which a user is located and motion data of a hand of the user; in a case where it is determined based on the environment data and the motion data that the motion of the hand of the user is limited by an obstacle, adjusting a direction of a virtual ray; and interacting with a virtual object based on the virtual ray after the direction is adjusted.

Description

Technical Field

[0001] This application relates to the field of extended reality technology, specifically to interaction methods and electronic devices. Background Technology

[0002] In the use of Extended Reality (XR) devices, ray-based interaction based on hand gestures is a widely used interaction method. Users can control the direction of the virtual ray through hand movements to select and manipulate objects in the virtual space.

[0003] In existing technologies, the direction of virtual rays depends entirely on the range of motion of the user's hand. When the range of motion of the user's hand is restricted by obstacles, the coverage of virtual rays is also limited, making it difficult for virtual rays to effectively hit the interactive target, resulting in interaction failure or reduced interaction efficiency. Summary of the Invention

[0004] The purpose of this application is to provide an interaction method and electronic device that can improve the success rate and efficiency of interacting with virtual objects.

[0005] In a first aspect, embodiments of this application provide an interaction method, which includes: acquiring environmental data of the user's physical environment and motion data of the user's hand; adjusting the direction of a virtual ray when it is determined based on the environmental data and the motion data that the user's hand movement is restricted by an obstacle; and interacting with a virtual object based on the virtual ray with the adjusted direction.

[0006] Secondly, embodiments of this application provide an interactive device, which includes: an acquisition unit for acquiring environmental data of the user's physical environment and motion data of the user's hand; an adjustment unit for adjusting the direction of a virtual ray when it is determined, based on the environmental data and the motion data, that the user's hand movement is restricted by an obstacle; and an interaction unit for interacting with a virtual object based on the virtual ray with the adjusted direction.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a computer program is stored, and when executed by a processor, the computer program implements the steps of the method described in the first aspect above.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, environmental data of the user's physical environment and motion data of the user's hand are first acquired. When it is determined that the user's hand movement is restricted by an obstacle based on these two types of data, the direction of the virtual ray is adjusted. Finally, the virtual ray with the adjusted direction interacts with the virtual object. Through this process, when the user's hand cannot continue to move due to an obstacle, the direction of the virtual ray can be adjusted to overcome the limitation of the hand's range of motion. This makes the ray's coverage no longer entirely dependent on the actual range of hand movement, reducing the occurrence of interrupted or failed interactive operations due to hand obstruction, thereby improving the success rate and efficiency of interaction with virtual objects. Attached Figure Description

[0012] Figure 1 This is a flowchart of the interaction method provided in the embodiments of this application; Figure 2 This is a schematic diagram illustrating an application scenario of the interaction method provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating an application scenario of the interaction method provided in the embodiments of this application; Figure 4 This is a schematic diagram illustrating an application scenario of the interaction method provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating an application scenario of the interaction method provided in the embodiments of this application; Figure 6 This is a schematic diagram illustrating an application scenario of the interaction method provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating an application scenario of the interaction method provided in the embodiments of this application; Figure 8 This is a schematic diagram illustrating an application scenario of the interaction method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the interactive device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0015] The interactive methods and devices provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0016] Please refer to Figure 1 This document illustrates one of the flowcharts of the interaction method provided in the embodiments of this application. The interaction method provided in the embodiments of this application can be applied to electronic devices. In practice, the aforementioned electronic device can be an extended reality device. Extended reality devices are a general term for hardware products capable of creating and presenting various human-computer interaction interfaces between completely real and completely virtual environments. Extended reality devices may include virtual reality (VR) devices, augmented reality (AR) devices, and mixed reality (MR) devices, etc. Examples include head-mounted displays, AR smart glasses, and MR headsets.

[0017] The interactive method provided in this application includes the following steps: Step 101: Obtain environmental data of the user's physical environment and motion data of the user's hands.

[0018] In this embodiment, the physical environment is the real three-dimensional space in which the user is located and the collection of all physical objects within it. For example, it may include the user's indoor room, physical objects such as tables, walls, chairs, and the floor within the room. The physical environment determines the actual movement boundaries of the user's hands.

[0019] Environmental data describes the physical environment in which a user is located. Specifically, it describes the three-dimensional spatial structure of the environment and the geometric properties of objects within it. This data is obtained through real-time scanning of the physical space around the user using a depth sensor in an extended reality device. In practice, environmental data includes spatial meshes and obstacle surface parameters. A spatial mesh represents the physical space as a discretized form of voxels or triangular patches, used to record the occupancy status of various locations in space. Obstacle surface parameters include the position coordinates and normal directions of the obstacle surfaces.

[0020] Motion data describes the movement and posture of a user's hand in three-dimensional space. Specifically, it may include, but is not limited to, the user's hand's position coordinates, direction of movement, velocity, and acceleration. Motion data can be acquired in real-time using hand-tracking sensors or cameras in extended reality devices.

[0021] As an example, a user wearing an MR headset sits at an office desk. The headset scans the surrounding environment using a built-in depth camera, generating spatial mesh data including the desk, walls, and floor, while simultaneously acquiring the world coordinate system position of the desktop and its upward normal vector. At the same time, the headset tracks the user's right hand using front-facing binocular vision cameras, acquiring real-time motion data such as the right hand's 3D spatial position and direction of movement.

[0022] Step 102: If it is determined that the user's hand movement is restricted by obstacles based on environmental data and motion data, adjust the direction of the virtual ray.

[0023] In this embodiment, an obstacle is a physical object in the user's physical environment that can restrict the range of physical movement of the user's hand. For example, a horizontal table in front of the user, a vertical wall to the side, etc. When an obstacle prevents the user's hand from continuing to move in a certain direction as originally intended, the user's hand movement can be considered restricted by the obstacle.

[0024] A virtual ray is a virtual straight line rendered by an extended reality device. A virtual ray has no physical entity and is only visible in virtual space. The direction of the virtual ray is geometrically related to the position or posture of the user's hand; the user can change the direction of the ray by moving their hand, thereby interacting with the virtual object they intend to interact with.

[0025] In this embodiment, it can be determined whether a user's hand movement is restricted by an obstacle based on environmental data and motion data. Typically, this restriction manifests as the hand being too close to the obstacle, a forced decrease in hand movement speed, or repetitive, tentative hand movements. Therefore, obstacle information, such as edge coordinates, in the direction of the user's hand movement can be extracted from the environmental data first. Then, the movement trend of the user's hand is determined by combining this information with the hand movement data. When the user's hand is too close to the obstacle and the movement trend is abnormal, such as deceleration or repeated small, reciprocating movements, it can be determined that the user's hand movement is restricted by an obstacle. By comprehensively determining scenarios where the user's hand movement is restricted by obstacles based on environmental data and hand movement data, it is possible to accurately identify situations where the user's interaction is hindered due to real-world physical obstacles, distinguishing between normal hand cessation and forced cessation due to obstacles, thus ensuring the accuracy of subsequent adjustments to the virtual ray direction.

[0026] When it is determined that the user's hand movement is restricted by an obstacle, the direction of the virtual ray can be adjusted. The adjusted ray can then point to areas that the hand could not originally reach due to the obstacle. Specific adjustment methods can include automatically shifting the virtual ray's emission point away from the obstacle, automatically changing the virtual ray's pointing angle based on the user's hand movement trend, or guiding the user to manually change the virtual ray's pointing angle, thus enabling the adjusted ray to point to areas that were previously inaccessible. By adjusting the direction of the virtual ray when the user's hand movement is determined to be restricted by an obstacle, the inherent binding relationship between the virtual ray's direction and the user's physical hand movement range in existing technologies is broken, so that the coverage area of ​​the virtual ray is no longer limited to the actual reachable area of ​​the hand.

[0027] Continuing the previous example, the user wants to click the virtual settings button located under the table, so they raise their right hand and move it downwards. The headset, based on environmental data, identifies the table directly in front of the user's hand as an obstacle, and simultaneously detects that the right hand is continuously moving towards the tabletop based on hand movement data. The headset determines that the user's downward hand movement will be restricted by the table, and then adjusts the direction of the virtual ray, shifting it downwards to point towards the area where the virtual settings button is located under the table.

[0028] Step 103: Interact with the virtual object based on the virtual ray after the direction is adjusted.

[0029] In this embodiment, a virtual object refers to an interactive element generated by the extended reality device and displayed in a virtual environment. For example, it can be a virtual interface element such as a virtual button, virtual menu, or virtual icon, or a virtual three-dimensional object such as a virtual model, virtual prop, or virtual character. Users can perform operations such as selection, triggering, and dragging with virtual objects using a virtual ray. For example, when a virtual ray hits or points at a virtual object, the virtual object can be highlighted, a click event can be triggered, or a drag or menu can be expanded.

[0030] Continuing with the previous example, the adjusted virtual ray hits the virtual settings button under the table. The headset recognizes that the button has been selected and then executes the operation to open the system settings interface. The user can successfully complete the interaction of clicking the virtual settings button without having to reach through the table.

[0031] The method provided in the above embodiments of this application first acquires environmental data of the user's physical environment and motion data of the user's hand. When it is determined that the user's hand movement is restricted by an obstacle based on these two types of data, the direction of the virtual ray is adjusted. Finally, the virtual ray with the adjusted direction interacts with the virtual object. Through this process, when the user's hand cannot continue to extend its movement due to an obstacle, the limitation on the hand's range of motion can be overcome by adjusting the direction of the virtual ray. This makes the ray's coverage area no longer entirely dependent on the actual range of hand movement, reducing the occurrence of interrupted or failed interactive operations due to hand obstruction, thereby improving the success rate and efficiency of interaction with virtual objects.

[0032] In some optional embodiments, after step 101 is performed, the following steps may also be performed to determine whether the user's hand movements are restricted by an obstacle: Step S11: Determine the direction of motion and acceleration of the user's hand based on the motion data.

[0033] The direction of motion refers to the vector direction of the user's hand displacement in three-dimensional space. Its function is to indicate the movement trend of the user's hand and help the device locate obstacles that may hinder the hand movement.

[0034] Acceleration refers to the rate of change of a user's hand movement velocity over time. The magnitude of acceleration indicates how fast the velocity changes, and the sign of acceleration indicates the direction of the velocity change. Acceleration is used to determine whether the hand is accelerating or decelerating. In practical applications, when the hand approaches an obstacle, it naturally decelerates; at this time, the acceleration is negative, meaning the direction of acceleration is opposite to the direction of movement.

[0035] In practice, the three-dimensional spatial coordinates of the same reference point on the user's hand in consecutive frames can be extracted from the motion data first. Then, the difference between the reference point's position coordinates in the current frame and the previous frame is calculated to obtain the displacement vector. Next, this displacement vector is divided by the frame interval to obtain the velocity vector. The direction of this velocity vector is the direction of hand movement.

[0036] Furthermore, the velocity vector difference between the current frame and the previous frame can be calculated to obtain the velocity change vector. Then, this velocity change vector is divided by the frame interval to obtain the acceleration vector. When the direction of the acceleration vector is opposite to the direction of the velocity vector, it indicates that the hand is in a deceleration state.

[0037] Step S12: Determine obstacles in the direction of movement based on environmental data.

[0038] An obstacle in the direction of movement refers to a physical entity located on the extension line of the user's hand movement direction, capable of preventing the hand from continuing to move. Examples include a horizontal table in the direction of downward hand movement, a vertical wall in the direction of forward hand movement, and the armrest of an office chair in the direction of leftward hand movement.

[0039] In practice, environmental data can include the position data and normal information of the surfaces of various physical objects in the user's physical environment. A ray can be emitted from the user's current hand position along the direction of hand movement, and the intersections of this ray with the surfaces of physical objects can be detected. The first physical object that intersects with the ray is identified as an obstacle in the direction of movement.

[0040] As an example, a user wearing an MR headset sits at an office desk. See also Figure 2 The headset extends the user's vertical downward hand movement direction into a ray, which intersects with the table surface in the environmental data. Therefore, the headset determines that the obstacle in the direction of the user's hand movement is the horizontal table surface in front of them.

[0041] Step S13: If at least one of the following conditions is met, it is determined that the user's hand movement is restricted by an obstacle: the distance between the user's hand and the obstacle is less than a first threshold and the acceleration is negative; the distance between the user's hand and the obstacle is less than a second threshold and the number of reciprocating movements of the user's hand within a preset time period is greater than a third threshold.

[0042] The following explanation is divided into two cases: In scenario one, if the distance between the user's hand and the obstacle is less than the first threshold and the acceleration is negative, it can be determined that the user's hand movement is restricted by the obstacle.

[0043] The distance between the user's hand and the obstacle refers to the shortest straight-line distance from a specified reference point on the user's hand to the surface of the obstacle. The obstacle surface can be either a plane or a curved surface. The specified reference point can be a wrist point or a palm point, etc., and is not specifically limited here. For an example, see [link to example]. Figure 2 The distance between a user's hand and an obstacle can be represented as the shortest straight-line distance from the user's wrist to the tabletop. It can be used to measure the proximity of the hand to the obstacle. The smaller the distance, the closer the hand is to the obstacle surface.

[0044] The first threshold is a pre-set maximum distance between the user's hand and an obstacle, used to trigger a hand movement restriction detection. Its function is to define the proximity range between the user's hand and the obstacle; when the distance between the user's hand and the obstacle is less than this value, the device begins to detect changes in hand acceleration. The first threshold can be adjusted according to different interaction scenarios and user habits; its specific value is not limited here. For example, it can be set to 5 cm for desktop interaction scenarios and 8 cm for wall interaction scenarios.

[0045] A negative acceleration value means that the direction of the hand's acceleration is opposite to the direction of hand movement; that is, the user's hand gradually decreases in speed as it moves towards the obstacle. For example, see... Figure 3 The curve showing the relationship between hand movement velocity and time is shown. In practice, the acceleration vector and the direction of motion vector can be used as a dot product. If the dot product result is less than 0, it indicates that the acceleration is negative.

[0046] Understandably, if the distance between the user's hand and the obstacle is less than a first threshold, it indicates that the hand is close to the obstacle, posing a risk of physical collision. Under this premise, a negative acceleration value indicates that the hand is decelerating, and this deceleration is usually caused by the user actively slowing down because they perceive their hand approaching the obstacle. When both conditions are met, it can be reasonably inferred that the hand's original intention to move is hindered by the obstacle, forcing the user to slow down to avoid a collision. At this point, the actual range of hand movement is limited by the spatial position of the obstacle, and the hand cannot continue to move freely in its original direction. Therefore, based on these two conditions, it can be reliably determined that the hand's movement is restricted by the obstacle, without waiting for the hand to actually touch the obstacle or completely stop moving.

[0047] In the second scenario, if the distance between the user's hand and the obstacle is less than the second threshold, and the number of reciprocating movements of the user's hand within a preset time period is greater than the third threshold, it can be determined that the user's hand movement is restricted by the obstacle.

[0048] The second threshold is a pre-set maximum distance between the user's hand and an obstacle, used to trigger reciprocating motion detection. When the distance between the user's hand and the obstacle is less than the second threshold, the device begins counting the number of reciprocating movements of the user's hand. The second threshold can be the same as or different from the first threshold. Optionally, the second threshold can be less than the first threshold to suit scenarios where the user's hand is already very close to the obstacle. Its specific value is not limited here.

[0049] A preset time period refers to a fixed-length time window used to observe whether reciprocating hand movements occur. For example, it could be within 2 seconds after detecting that the distance between the hand and the obstacle is less than a second threshold. Within the preset time period, the number of reciprocating hand movements can be recorded.

[0050] Reciprocating motion refers to the back-and-forth movement of the hand. It typically occurs when the hand has already touched or is extremely close to an obstacle, and the user attempts to maneuver around it with small adjustments. Reciprocating motion can be the back-and-forth swinging of the hand at a specified reference point, or the back-and-forth swinging of the wrist angle. Specifically, a single change in wrist angle exceeding a preset angle threshold, with the direction of change within a preset angle range, can be counted as a valid reciprocating motion. Alternatively, a change in the position of the hand's specified reference point exceeding a preset distance threshold, with the direction of change within a preset angle range, can also be counted as a valid reciprocating motion.

[0051] The third threshold is a pre-set minimum number of reciprocating movements used to trigger a restriction determination. Its function is to distinguish between normal hand tremors and repeated attempts caused by obstruction. When the number of reciprocating movements within a preset time period exceeds this value, it is determined that the user's hand movement is restricted by an obstacle. The specific value of this threshold is not limited here.

[0052] Understandably, if the distance between the user's hand and the obstacle is less than the second threshold, it indicates that the hand has entered an area extremely close to the obstacle. At this point, the remaining space between the hand and the obstacle surface is very small. (See [link to relevant documentation]). Figure 6 As shown. In normal, unobstructed movement, the hand moves continuously in one direction without frequently changing direction within a very small space. When a user repeatedly changes the direction of hand position or joint angle within a short period of time, and each change is very small, for example... Figure 7The curve showing the relationship between wrist angle and time indicates that the user is not moving normally in free space, but rather attempting to break through some physical boundary. Therefore, the high-frequency, small-amplitude reciprocating movements of the hand while close to the obstacle represent the user's behavior when attempting to cross the obstacle but being physically blocked. When both conditions are met simultaneously, it can be reasonably inferred that the actual hand movement is restricted by the obstacle's surface, and the user is trying to find an operable ray direction or regain interactive capabilities through repeated trial and error. Based on this, it can be determined that the hand movement is restricted by the obstacle.

[0053] This embodiment determines the direction and acceleration of the user's hand movement based on motion data, enabling real-time monitoring of the hand's movement trend and speed changes. Simultaneously, it identifies obstacles in the direction of movement based on environmental data, pinpointing specific physical entities that will obstruct the hand. Furthermore, by detecting the combination of a distance less than a first threshold and negative acceleration between the user's hand and the obstacle, it accurately identifies the hand's deceleration due to approaching the obstacle, thus confirming that the hand's movement is restricted by the obstacle. By detecting a distance less than a second threshold and a number of hand reciprocating movements greater than a third threshold within a preset time period, it identifies a state where the hand is unable to move forward near the obstacle and the user is attempting to overcome the restriction through repeated small probes, thus confirming that the hand's movement is restricted by the obstacle. Therefore, it can perceive the restriction state before the hand actually touches the obstacle, avoiding reliance on passive detection after contact, achieving timely and accurate identification of the obstructed state. This provides a reliable trigger condition for subsequent adjustment of the virtual ray direction, helping to reduce false positives and false negatives.

[0054] In some optional embodiments, the direction of the virtual ray is from the direction control point of the virtual ray towards the user's hand. In step 102 above, the direction of the virtual ray can be further adjusted by moving the direction control point away from the obstacle.

[0055] A virtual ray's direction control point is a spatial point used to define the direction of the virtual ray. It can be a preset spatial point, and its direction, along with the user's hand position, determines the virtual ray's direction. Specifically, the direction of the virtual ray is a vector pointing from the direction control point to the user's hand position. The user's hand position is the coordinate of a specified reference point in three-dimensional space, and its direction, along with the direction control point, determines the virtual ray's direction. This position can be obtained from the aforementioned motion data.

[0056] The direction control point does not participate in ray collision detection or emission; it serves only as a geometric reference point. When direction adjustment is not triggered, the direction control point is located at a preset offset or fixed position behind the user's hand, meaning the user's hand is positioned between the direction control point and the virtual object, causing the virtual ray to appear to originate near the hand and point away. After direction adjustment is triggered, the direction control point can move along a specified direction, thereby changing the direction of the virtual ray, while the user's hand position remains unchanged. The first position is the position of the direction control point when no virtual ray direction adjustment is performed.

[0057] Moving the direction control point away from obstacles means translating its spatial position in a direction perpendicular to the obstacle's surface and away from it. Since the direction of the virtual ray is defined as pointing from the direction control point towards the user's hand, changes in the direction control point's position will directly alter the ray's direction. Moving away from obstacles allows the adjusted ray to bypass them and reach areas that were previously inaccessible due to the hand's obstruction.

[0058] By defining the direction of the virtual ray as pointing from the direction control point to the user's hand position, a geometric relationship is established between the direction control point and the ray's direction. This means that changes in the position of the direction control point directly determine the ray's direction. Moving the direction control point away from obstacles moves it from its original position to a new position outside the obstacle. Since the virtual ray always points from the direction control point to the user's hand position, changes in the direction control point's position cause the ray's direction to deflect, effectively changing the ray's direction without altering the actual hand position. Therefore, when the user's hand is unable to continue moving in the desired direction due to physical obstacles, moving the direction control point automatically deflects the virtual ray to cover the previously obstructed target area. This restores the continuity and accessibility of the interaction without increasing the user's operational burden, improving the success rate and efficiency of interacting with virtual objects.

[0059] In some optional embodiments, the direction of the virtual ray can be further adjusted by the following steps: Step S21: Determine the direction of movement of the direction control point, which is the direction away from the obstacle.

[0060] The direction of movement of a directional control point refers to the geometric direction along which the directional control point translates in space. This direction is defined as moving away from obstacles, that is, pushing the directional control point from the side closer to the obstacle to the side farther away from the obstacle. The purpose of controlling the direction of movement is to ensure that the movement of the directional control point does not move further closer to the obstacle, but rather towards a spatial area that can compensate for the limitations of hand movement.

[0061] The direction away from an obstacle refers to the direction from the obstacle's surface towards the external space. For planar obstacles, this direction is the outward direction of the normal to the obstacle's surface. For example, when the obstacle is a horizontal tabletop, the movement direction can be vertically upward; when the obstacle is a vertical wall, the movement direction is horizontal, away from the wall. For curved obstacles, this direction is the outward direction of the normal to the tangent plane at the point where the user's hand is closest to the obstacle's surface. The purpose of this direction is to move the directional control point in the opposite direction to the restricted hand movement, thereby compensating for insufficient ray coverage caused by the hand's inability to move further.

[0062] Step S22: Determine the movement distance of the direction control point based on the acceleration of the user's hand and the distance between the user's hand and the obstacle.

[0063] Movement distance refers to the length of the virtual ray direction control point moved along the movement direction. Movement distance is a non-negative scalar, and its magnitude depends on the user's hand acceleration and the distance between the hand and the obstacle. The function of movement distance is to control the degree of adjustment of the virtual ray direction.

[0064] In practice, the movement distance of the directional control point can be calculated based on a pre-defined mathematical formula. This mathematical formula can be expressed as an empirical formula, a mapping table, or a trained regression model. Specifically, the currently acquired user hand acceleration value and the distance between the user's hand and the obstacle can be used as input parameters, substituted into the mathematical formula, and a scalar value is output as the movement distance. This mathematical formula must ensure that the larger the absolute value of the acceleration, the larger the output movement distance; and the smaller the distance between the user's hand and the obstacle, the larger the output movement distance.

[0065] Understandably, acceleration reflects the degree of deceleration of the user's hand. In scenarios approaching obstacles, the more severe the deceleration and the greater the absolute value of the acceleration, the faster the user's intended movement, or the more eager the user is to complete the operation before hitting the obstacle. Therefore, a larger ray direction adjustment is needed to compensate for the obstructed range of motion. The distance between the user's hand and the obstacle reflects the current spatial proximity of the hand to the obstacle. The closer the hand is to the obstacle, the smaller its remaining space for movement, and the greater the actual restriction on the virtual ray. Therefore, a larger adjustment range is needed to overcome the obstacle and point to the interactive target. Combining acceleration and distance allows us to capture both the intensity of the user's intention and quantify the degree of restriction, thereby reasonably determining the distance the ray direction control point should move, ensuring that the adjusted ray effectively covers the user's desired interactive area.

[0066] Step S23: Move the direction control point according to the direction and distance of movement.

[0067] Understandably, see Figure 5 Before adjustment, the virtual ray's direction points from the first position towards the user's hand. When the user's hand is near an obstacle, the hand's movement is limited to above the obstacle's surface, preventing the user from reaching below. If the first position remains unchanged, the ray's direction is restricted by the hand's position, only pointing above the obstacle and unable to reach the lower interactive target obscured by the obstacle. When the direction control point is moved along the aforementioned direction, it moves upward from the first position to the second position. The ray's direction from the second position towards the user's hand changes. Because the direction control point rises while the hand position remains essentially unchanged, the adjusted virtual ray's direction deflects downward relative to the original direction. This deflection allows the virtual ray to cross the obstacle's edge and point towards the space below, effectively projecting the virtual ray to a lower position without moving the hand. Therefore, the original coverage area of ​​the virtual ray is expanded, allowing the ray to reach the interactive area previously obscured by the obstacle.

[0068] This embodiment determines the direction of movement of the directional control point to be away from obstacles, allowing the control point to move towards a spatial area that compensates for hand movement limitations. The movement distance is determined based on acceleration and the distance between the user's hand and the obstacle, thus correlating the adjustment range with the degree of deceleration of the user's hand and the remaining hand movement space. By moving the directional control point according to the direction and distance, effective updates to the directional control point's position are achieved. Therefore, the determination of the movement distance considers both the intensity of the user's intent and the degree of limitation, adapting the ray adjustment range to the actual situation, improving the rationality of the ray direction adjustment, avoiding under-adjustment or over-adjustment caused by fixed adjustment ranges, and thereby improving the continuity and success rate of the interaction.

[0069] In some optional embodiments, step S21 above may further include the following steps: determining the point on the obstacle surface that is closest to the user's hand; pointing the closest point toward the user's hand, and determining it as the direction of movement of the direction control point.

[0070] The closest point is the point on the obstacle surface that has the smallest spatial distance from the user's hand-designated reference point. For planar obstacles, this point is the foot of the perpendicular from the user's hand to the plane. For curved obstacles, this point lies on the curved surface, and the line connecting the user's hand to this point is perpendicular to the tangent plane of the curved surface at that point.

[0071] The direction pointing towards the user's hand refers to the vector direction originating from the nearest point on the obstacle surface and heading towards the specified reference point on the user's hand. Moving the direction control point along this direction moves it away from the obstacle from its original position, thus adjusting the direction of the virtual ray.

[0072] By automatically determining the movement direction of the direction control point by the geometric relationship between the obstacle surface and the user's hand, the direction control point can always move away from the obstacle, thus ensuring the accuracy and consistency of the ray direction adjustment, while improving the versatility of the solution in obstacle scenarios of different shapes.

[0073] In some optional embodiments, step S22 above may further include the following steps: Step S31: Determine the ratio of the distance between the user's hand and the obstacle to a first threshold.

[0074] Specifically, the ratio of the two is a number between 0 and 1. The smaller the ratio, the closer the hand is to the obstacle; the closer the ratio is to 1, the farther the hand is from the first threshold boundary.

[0075] Step S32: Based on the ratio, determine the first coefficient, which is negatively correlated with the ratio. The first coefficient is used to indicate the proximity of the user's hand to the obstacle.

[0076] The first coefficient is a dimensionless value. It is negatively correlated with the ratio mentioned above; that is, the smaller the ratio, the larger the first coefficient, and vice versa. The first coefficient characterizes the proximity of the user's hand to the obstacle, serving as a factor for adjusting the movement distance. Proximity refers to the spatial closeness between the user's hand and the obstacle surface; a higher proximity indicates a closer hand to the obstacle, requiring a larger adjustment range in the ray direction. In practice, the difference between 1 and the ratio mentioned above can be determined as the first coefficient. That is, the first coefficient can be expressed as (1 - d / D) threshold ).

[0077] Step S33: Determine the acceleration component in the opposite direction to the direction of movement.

[0078] The acceleration component in the aforementioned direction of movement refers to the projection of the user's hand acceleration vector onto that direction. Since the hand typically decelerates as it approaches an obstacle, the acceleration vector direction is opposite to the direction of movement, which is usually towards the obstacle; therefore, the acceleration component is typically negative. The absolute value of the acceleration component reflects the degree of hand deceleration.

[0079] Step S34: Based on the acceleration component and the first coefficient, determine the movement distance of the direction control point. The movement distance is proportional to the absolute value of the acceleration component and the first coefficient, respectively.

[0080] Here, the distance traveled can be denoted as... It can be calculated using the following formula:

[0081] Where k is the adjustment coefficient, which can be preset as needed; The above acceleration components are represented by d; d is the distance between the user's hand and the obstacle; D threshold This is the first threshold.

[0082] This embodiment quantifies the spatial proximity of the hand to the obstacle by determining the ratio of the distance between the user's hand and the obstacle to a first threshold, and then determining a first coefficient negatively correlated with this ratio. By determining the acceleration component of the acceleration in the direction of movement of the virtual control point, the severity of hand deceleration is quantified, reflecting the intensity of the user's intention to cross the obstacle. By determining the movement distance based on the acceleration component and the first coefficient, and making the movement distance directly proportional to the absolute value of the acceleration component and the first coefficient, a positive correlation is achieved between the adjustment range and the intensity of the user's intention and the degree of constraint. This means that the more abrupt the user's deceleration and the closer the hand is to the obstacle, the greater the movement distance of the directional control point, and the more pronounced the virtual ray deflection. Therefore, the determination of the movement distance more closely matches the user's actual operational intention, avoiding under-adjustment or over-adjustment that may result from a fixed adjustment range, thereby improving the accuracy of ray adjustment.

[0083] In some optional embodiments, in step 102 above, the direction of the virtual ray can be further adjusted by the following steps: Step S51: Display the virtual control, which is used to control the direction of the virtual ray.

[0084] Virtual controls are visual interactive components generated in an extended reality environment. Virtual controls do not have a physical entity; they are displayed graphically at specific locations within the user's field of vision, such as above the user's hand, near the wrist, or a fixed area in space. The style of virtual controls is not limited; for example, it could be a semi-transparent circular joystick or a long, rectangular slider, hovering about 2 centimeters above the back of the hand.

[0085] Virtual controls can be used to control the direction of a virtual ray. Specifically, user actions on virtual controls, such as sliding and dragging, translate into adjustments to the pitch or yaw angle of the virtual ray in space. The mapping relationship between virtual control operations and virtual ray direction changes is preset; for example, moving a virtual control up or down corresponds to the vertical deflection of the virtual ray, and moving a virtual control left or right corresponds to the horizontal deflection of the virtual ray.

[0086] Step S52: Receive the user's first input to the virtual control.

[0087] The first input can be used to control the movement of the virtual control. The first input can be touch input, voice command, a specific gesture entered by the user, or other feasible input. The specific input can be determined according to actual usage needs, and this application embodiment does not limit it.

[0088] As an example, see Figure 8 After the user repeatedly probes the desktop with their hand, and the system detects multiple back-and-forth hand movements, a semi-transparent rectangular slider (as shown in label 801) appears 2 centimeters above the user's hand. This rectangular slider is the virtual control. The user can complete the first input by sliding their finger across the rectangular slider.

[0089] In step S53, in response to the first input, the virtual control is moved, and the direction of the virtual ray is adjusted based on the movement direction and distance of the virtual control.

[0090] The movement direction of a virtual control is the vector direction of its displacement in virtual space when dragged by the user. The movement distance of a virtual control is the linear length by which it moves from its initial position along the movement direction, and can be used to control the adjustment range of the virtual ray direction. The movement distance of the virtual control is positively correlated with the adjustment range of the virtual ray direction; this correspondence can be preset and stored in the device for later retrieval.

[0091] Continuing the previous example, when the user slides the rectangular slider from position 801 to position 802, the virtual ray deflects downwards. This deflection allows the virtual ray to cross the edge of the obstacle surface and point to the space below the obstacle, effectively projecting the virtual ray to a lower position. Therefore, it breaks through the original coverage area of ​​the virtual ray, allowing the ray to reach the interactive area that was originally obscured by the obstacle.

[0092] This embodiment provides a visual adjustment interface for the user by displaying a virtual control for controlling the direction of the virtual ray. By receiving the user's first input to the virtual control, the system responds to the input by moving the virtual control and adjusting the direction of the virtual ray based on its movement direction and distance, directly mapping the user's intuitive dragging action into a change in the ray's angle. Therefore, when the user's hand is restricted by physical obstacles and cannot change the ray's direction through natural movement, the user can manually adjust the ray's direction by manipulating the virtual control, directing it to the desired interactive area. This embodiment provides an alternative method for adjusting the virtual ray's direction besides automatic adjustment. It is suitable for scenarios where the hand is already pressed against an obstacle and automatic adjustment is limited. It allows for precise control of the virtual ray's direction through active user intervention, thereby improving the flexibility and success rate of interaction with virtual objects.

[0093] In some optional embodiments, step S53 above may further include the following steps: Step S61: When the virtual control moves vertically, determine the vertical movement distance of the virtual control, and adjust the pitch angle of the virtual ray based on the vertical movement distance.

[0094] Vertical movement refers to the movement of a virtual control along the vertical direction in screen space. The vertical movement distance is the linear displacement of the virtual control relative to its initial position in the vertical direction. The vertical movement distance can be calculated by tracking the difference between the hand's initial position and its current position on the vertical axis.

[0095] The pitch angle refers to the angle of a virtual ray relative to the horizontal direction in the vertical plane. The horizontal direction is usually defined as 0 degrees, upward is positive, and downward is negative. The purpose of the pitch angle is to control the direction of the ray in the vertical direction; changes in the pitch angle allow the virtual ray to swing up and down, pointing towards virtual objects at different heights.

[0096] In practice, the longitudinal movement distance can be proportional to the change in the pitch angle of the virtual ray, and the proportional coefficient between the two can be preset as needed. Here, the specific value of the coefficient is not limited.

[0097] Step S62: When the virtual control moves laterally, determine the lateral movement distance of the virtual control and adjust the yaw angle of the virtual ray based on the lateral movement distance.

[0098] Lateral movement refers to the horizontal movement of a virtual control within screen space. The lateral movement distance is the linear displacement of the virtual control relative to its initial position in the horizontal direction. This distance can be calculated by tracking the difference between the hand's initial position and its current position on the horizontal axis.

[0099] Yaw angle refers to the angle of a virtual ray relative to the front in the horizontal plane. It is usually defined as 0 degrees to the front, positive to the right, and negative to the left. The purpose of the yaw angle is to control the direction of the ray in the horizontal direction; its changes allow the ray to swing left and right, pointing towards virtual objects at different horizontal positions.

[0100] In practice, the change in the lateral movement distance and the yaw angle of the virtual ray can be directly proportional. The proportional coefficient between the two can be preset as needed, and its specific value is not limited here.

[0101] This embodiment determines the vertical movement distance of the virtual control and adjusts the pitch angle of the virtual ray based on that distance, directly mapping the user's vertical swiping motion to the vertical swinging effect of the virtual ray. Similarly, by determining the horizontal movement distance of the virtual control and adjusting the yaw angle of the virtual ray based on that distance, the user's left-right swiping motion is mapped to the left-right swinging effect of the virtual ray. Therefore, users can independently control the pitch and yaw angles of the ray by dragging along different directions on a single virtual control, achieving full-range adjustment. Compared to automatic adjustment of the virtual ray's direction, manual adjustment allows users to arbitrarily change the ray's direction according to actual needs, making it particularly suitable for scenarios where automatic adjustment cannot fully meet requirements or where users want to cover multiple targets in different directions, thus improving the flexibility of interaction with virtual objects.

[0102] In some alternative embodiments, the virtual ray can be displayed in a first style before its direction is adjusted; and in a second style after its direction is adjusted, the second style differs from the first style in one or more visual attributes.

[0103] The first style is the default display style of the virtual ray when no adjustment mechanism is triggered. This style can include visual attributes such as the virtual ray's color, transparency, line type, glow effect, width, or texture. The purpose of the first style is to allow the user to recognize the normal state of the virtual ray, indicating that the current ray direction completely follows the natural movement of the hand without any compensation intervention. For example, the first style is a blue, semi-transparent, solid line beam with a width of 2 pixels.

[0104] The second style is the display style of the virtual ray after the direction adjustment mechanism is triggered. The second style differs from the first style in one or more visual attributes, such as changing the color, changing the line type, adding an outer glow, or adding an arrow marker. The purpose of the second style is to indicate to the user that the current ray direction has been adjusted. For example, the second style is an orange, semi-transparent, 2-pixel-wide dashed beam with a periodic blinking effect.

[0105] Furthermore, if it is determined based on environmental and motion data that the user's hand movements are not restricted by obstacles, the direction of the virtual ray can be restored to its original direction and re-rendered and displayed in the first style.

[0106] This embodiment presents the normal appearance of a virtual ray to the user by displaying it in a first style before adjusting its direction; and by displaying it in a second style after adjusting its direction, it provides perceptible visual feedback when the ray's direction changes. Since the first and second styles differ significantly in at least one visual attribute, the user can intuitively perceive the change in the ray's state without additional operation or prompts. Therefore, when the user actively adjusts the ray's direction due to detected hand obstruction, they can understand that they are currently in a compensatory interaction mode through the change in the ray's style, avoiding confusion caused by the ray's direction not matching the hand's natural orientation. This improves the intuitiveness and transparency of the interaction and reduces user errors.

[0107] It should be noted that the interactive method provided in this application can be executed by an interactive device. This application uses an interactive device executing the interactive method as an example to illustrate the interactive device provided in this application.

[0108] like Figure 9 As shown, the interactive device 900 of this embodiment includes: an acquisition unit 901, used to acquire environmental data of the physical environment in which the user is located and motion data of the user's hand; an adjustment unit 902, used to adjust the direction of the virtual ray when it is determined based on the environmental data and the motion data that the user's hand movement is restricted by an obstacle; and an interaction unit 903, used to interact with virtual objects based on the virtual ray after the direction is adjusted.

[0109] In some optional implementations of this embodiment, the device further includes a determining unit, configured to: determine the direction of motion and acceleration of the user's hand based on the motion data; determine obstacles in the direction of motion based on the environmental data; and determine that the user's hand movement is restricted by obstacles if at least one of the following conditions is met: the distance between the user's hand and the obstacle is less than a first threshold, and the acceleration is negative; the distance between the user's hand and the obstacle is less than a second threshold, and the number of reciprocating movements of the user's hand within a preset time period is greater than a third threshold.

[0110] In some optional implementations of this embodiment, the direction of the virtual ray is from the direction control point of the virtual ray to the user's hand; the adjustment unit 902 is further configured to: move the direction control point away from the obstacle to adjust the direction of the virtual ray.

[0111] In some optional implementations of this embodiment, the adjustment unit 902 is further configured to: determine the movement direction of the direction control point, wherein the movement direction is a direction away from the obstacle; determine the movement distance of the direction control point based on the acceleration of the user's hand and the distance between the user's hand and the obstacle; and move the direction control point according to the movement direction and the movement distance.

[0112] In some optional implementations of this embodiment, the adjustment unit 902 is further configured to: determine the point on the surface of the obstacle that is closest to the user's hand; and determine the direction in which the closest point points to the user's hand as the movement direction of the direction control point.

[0113] In some optional implementations of this embodiment, the adjustment unit 902 is further configured to: determine the ratio of the distance between the user's hand and the obstacle to a first threshold; determine a first coefficient based on the ratio, the first coefficient being negatively correlated with the ratio, the first coefficient being used to indicate the degree of proximity between the user's hand and the obstacle; determine the acceleration component of the acceleration in the opposite direction to the direction of movement; and determine the movement distance of the direction control point based on the acceleration component and the first coefficient, the movement distance being proportional to the absolute value of the acceleration component and the first coefficient, respectively.

[0114] In some optional implementations of this embodiment, the adjustment unit 902 is further configured to: display a virtual control, the virtual control being used to control the direction of the virtual ray; receive a first input from the user to the virtual control; respond to the first input, control the virtual control to move, and adjust the direction of the virtual ray based on the moving direction and moving distance of the virtual control.

[0115] In some optional implementations of this embodiment, the adjustment unit 902 is further configured to: determine the longitudinal movement distance of the virtual control when the virtual control moves longitudinally, and adjust the pitch angle of the virtual ray based on the longitudinal movement distance; and determine the lateral movement distance of the virtual control when the virtual control moves laterally, and adjust the yaw angle of the virtual ray based on the lateral movement distance.

[0116] In some optional implementations of this embodiment, the device further includes a display unit for: displaying the virtual ray in a first style before adjusting the direction of the virtual ray; and displaying the virtual ray in a second style after adjusting the direction of the virtual ray, wherein the second style differs from the first style in one or more visual attributes.

[0117] The apparatus provided in the above embodiments of this application first acquires environmental data of the user's physical environment and motion data of the user's hand. When it is determined that the user's hand movement is restricted by an obstacle based on these two types of data, the direction of the virtual ray is adjusted. Finally, the virtual ray with the adjusted direction interacts with the virtual object. Through this process, when the user's hand cannot continue to extend its movement due to an obstacle, the limitation on the hand's range of motion can be overcome by adjusting the direction of the virtual ray. This makes the ray coverage no longer entirely dependent on the actual range of hand movement, reducing the occurrence of interrupted or failed interactive operations due to hand obstruction, thereby improving the success rate and efficiency of interaction with virtual objects.

[0118] The interactive device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0119] The interactive device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0120] The interactive device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0121] Optionally, such as Figure 10As shown, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described interaction method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0122] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0123] Figure 11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. The electronic device 1100 includes, but is not limited to, components such as: radio frequency unit 1101, network module 1102, audio output unit 1103, input unit 1104, sensor 1105, display unit 1106, user input unit 1107, interface unit 1108, memory 1109, and processor 1110.

[0124] Those skilled in the art will understand that the electronic device 1100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here. The processor 1110 is used to acquire environmental data of the user's physical environment and motion data of the user's hand; when it is determined that the user's hand movement is restricted by obstacles based on the environmental data and the motion data, the direction of the virtual ray is adjusted; and the virtual ray with the adjusted direction interacts with the virtual object.

[0125] Through the above process, when the user's hand cannot continue to move due to obstacles, the direction of the virtual ray can be adjusted to overcome the limitation of the hand's range of motion. This makes the ray coverage no longer completely dependent on the actual range of hand movement, reducing the occurrence of interruptions or failures in interactive operations due to hand obstruction, thereby improving the success rate and efficiency of interacting with virtual objects.

[0126] In some optional implementations of this embodiment, the processor 1110 is further configured to determine the direction and acceleration of the user's hand movement based on the motion data; determine obstacles in the direction of movement based on the environmental data; and determine that the user's hand movement is restricted by obstacles if at least one of the following conditions is met: the distance between the user's hand and the obstacle is less than a first threshold and the acceleration is negative; the distance between the user's hand and the obstacle is less than a second threshold and the number of reciprocating movements of the user's hand within a preset time period is greater than a third threshold.

[0127] In some optional implementations of this embodiment, the direction of the virtual ray is from the direction control point of the virtual ray to the user's hand; the processor 1110 is further configured to move the direction control point away from the obstacle to adjust the direction of the virtual ray.

[0128] In some optional implementations of this embodiment, the processor 1110 is further configured to determine the movement direction of the direction control point, wherein the movement direction is a direction away from the obstacle; determine the movement distance of the direction control point based on the acceleration of the user's hand and the distance between the user's hand and the obstacle; and move the direction control point according to the movement direction and the movement distance.

[0129] In some optional implementations of this embodiment, the processor 1110 is further configured to determine the point on the surface of the obstacle that is closest to the user's hand; and to determine the direction in which the closest point points to the user's hand as the movement direction of the direction control point.

[0130] In some optional implementations of this embodiment, the processor 1110 is further configured to: determine the ratio of the distance between the user's hand and the obstacle to a first threshold; determine a first coefficient based on the ratio, the first coefficient being negatively correlated with the ratio, the first coefficient being used to indicate the degree of proximity between the user's hand and the obstacle; determine the acceleration component of the acceleration in the opposite direction to the direction of movement; and determine the movement distance of the direction control point based on the acceleration component and the first coefficient, the movement distance being proportional to the absolute value of the acceleration component and the first coefficient, respectively.

[0131] In some optional implementations of this embodiment, the processor 1110 is further configured to display a virtual control via the display unit 1106, the virtual control being used to control the direction of the virtual ray; receive a first input from the user to the virtual control via the display unit 1106; and, in response to the first input, control the virtual control to move, and adjust the direction of the virtual ray based on the moving direction and moving distance of the virtual control.

[0132] In some optional implementations of this embodiment, the processor 1110 is further configured to, when the virtual control moves longitudinally, determine the longitudinal movement distance of the virtual control and adjust the pitch angle of the virtual ray based on the longitudinal movement distance; and when the virtual control moves laterally, determine the lateral movement distance of the virtual control and adjust the yaw angle of the virtual ray based on the lateral movement distance.

[0133] In some optional implementations of this embodiment, the processor 1110 is further configured to display the virtual ray in a first style via the display unit 1106; after adjusting the direction of the virtual ray, the method further includes: displaying the virtual ray in a second style via the display unit 1106, wherein the second style differs from the first style in one or more visual attributes.

[0134] It should be understood that, in this embodiment, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042. The GPU 11041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1106 may include a display panel 11061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0135] The memory 1109 can be used to store software programs and various data. The memory 1109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1109 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0136] Processor 1110 may include one or more processing units; optionally, processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1110.

[0137] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described interaction method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0138] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0139] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described interaction method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0140] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0141] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described interactive method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0142] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0144] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An interaction method, characterized in that, Applied to extended reality devices, the method includes: Acquire environmental data of the user's physical environment and the user's hand movement data; If, based on the environmental data and the motion data, it is determined that the user's hand movements are restricted by obstacles, the direction of the virtual ray is adjusted. The virtual ray, after its direction has been adjusted, interacts with the virtual object.

2. The method according to claim 1, characterized in that, After acquiring environmental data of the user's physical environment and motion data of the user's hand, the method further includes: The direction and acceleration of the user's hand movement are determined based on the motion data; Obstacles in the direction of movement are determined based on the environmental data; A user's hand movement is determined to be restricted by an obstacle if at least one of the following conditions is met: The distance between the user's hand and the obstacle is less than a first threshold, and the acceleration is negative; The distance between the user's hand and the obstacle is less than a second threshold, and the number of reciprocating movements of the user's hand within a preset time period is greater than a third threshold.

3. The method according to claim 1, characterized in that, The direction of the virtual ray is from the direction control point of the virtual ray towards the user's hand; adjusting the direction of the virtual ray includes: Move the direction control point away from the obstacle to adjust the direction of the virtual ray.

4. The method according to claim 3, characterized in that, Moving the direction control point away from the obstacle includes: Determine the direction of movement of the direction control point, wherein the direction of movement is away from the obstacle; The movement distance of the direction control point is determined based on the acceleration of the user's hand and the distance between the user's hand and the obstacle; The direction control point is moved according to the direction of movement and the distance of movement.

5. The method according to claim 4, characterized in that, Determining the movement direction of the direction control point includes: Determine the point on the surface of the obstacle that is closest to the user's hand; The direction in which the nearest point points toward the user's hand is determined as the movement direction of the direction control point.

6. The method according to claim 1, characterized in that, Adjusting the direction of the virtual ray includes: Display virtual controls, which are used to control the direction of the virtual ray; Receive the user's first input to the virtual control; In response to the first input, the virtual control is controlled to move, and the direction of the virtual ray is adjusted based on the movement direction and distance of the virtual control.

7. The method according to claim 6, characterized in that, Adjusting the direction of the virtual ray based on the movement direction and distance of the virtual control includes: When the virtual control moves vertically, the vertical movement distance of the virtual control is determined, and the pitch angle of the virtual ray is adjusted based on the vertical movement distance; When the virtual control moves laterally, the lateral movement distance of the virtual control is determined, and the yaw angle of the virtual ray is adjusted based on the lateral movement distance.

8. The method according to claim 1, characterized in that, Prior to adjusting the direction of the virtual ray, the method further includes: The virtual ray is displayed in the first style; After adjusting the direction of the virtual ray, the method further includes: The virtual ray is displayed in a second style, which differs from the first style in one or more visual attributes.

9. An interactive device, characterized in that, The device includes: The acquisition unit is used to acquire environmental data of the user's physical environment and the movement data of the user's hands; An adjustment unit is used to adjust the direction of the virtual ray when it is determined, based on the environmental data and the motion data, that the user's hand movement is restricted by an obstacle; An interaction unit is used to interact with virtual objects based on the virtual ray after its direction has been adjusted.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the interactive method as described in any one of claims 1-8.

Images