Interaction method, apparatus and electronic device for extended reality scene

Abstract

The method comprises the following steps: according to the motion control operation of a controller on a target object in an extended reality scene, controlling the target object to move based on a target movement track, wherein the target movement track is determined by a movement track of the controller; in response to detecting that a current displacement of the controller is greater than a preset distance threshold, determining a first movement speed according to the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by a current position of the controller and an initial position of the controller, and the initial position of the controller is a position of the controller when the controller starts to control the target object to move; and controlling the target object to move based on the first movement speed. The control of the object in the extended reality scene in the above interaction manner conforms to the operation habit of a user, and improves the interaction efficiency of the user and the object in the extended reality scene.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an interactive method, apparatus, and electronic device for extending real-world scenarios. Background Technology

[0002] Because of its immersive, interactive, and imaginative features, extended reality technology is increasingly being applied in various industries, such as gaming and home renovation.

[0003] In extended reality scenarios, users can interact with objects remotely, such as moving objects within the extended reality scene.

[0004] The inventors discovered that the current methods for manipulating objects in extended reality scenarios are not intuitive, resulting in low efficiency for users interacting with objects in extended reality scenarios. Summary of the Invention

[0005] This disclosure provides an interactive method, apparatus, and electronic device for extended reality scenarios, overcoming the problem in related technologies where the operation of objects in extended reality scenarios is not intuitive, resulting in low interaction efficiency between users and objects in extended reality scenarios.

[0006] In a first aspect, embodiments of this disclosure provide an interactive method for an extended reality scene, the method comprising: controlling the target object to move based on a target movement trajectory according to a controller's motion control operation on a target object in the extended reality scene; wherein the target movement trajectory is determined by the movement trajectory of the controller; in response to detecting that the current displacement of the controller is greater than a preset distance threshold, determining a first movement speed based on the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, the initial position of the controller being the position of the controller when the controller begins to control the movement of the target object; and controlling the target object to move based on the first movement speed.

[0007] Secondly, embodiments of this disclosure provide an interactive device for extended reality scenarios, the device comprising: a first control unit, configured to control the target object to move based on a target movement trajectory according to a controller's motion control operation on a target object in the extended reality scenario; wherein the target movement trajectory is determined by the movement trajectory of the controller; a determining unit, configured to determine a first movement speed based on the controller's current displacement and the preset distance threshold in response to detecting that the controller's current displacement is greater than a preset distance threshold, wherein the current displacement is determined by the controller's current position and the controller's initial position, the controller's initial position being the position of the controller when the controller begins to control the target object to move; and a second control unit, configured to control the target object to move based on the first movement speed.

[0008] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the at least one processor to perform the first aspect above and various possible interactive methods for extending real-world scenarios involved in the first aspect.

[0009] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the first aspect above and various possible interactive methods for extending real-world scenarios.

[0010] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect above and various possible interactive methods for extending real-world scenarios.

[0011] The interactive method, device, and electronic device for extended reality scenarios provided in this embodiment control the target object to move based on a target trajectory according to the motion control operation of the controller on the target object in the extended reality scenario; wherein, the target movement trajectory is determined by the movement trajectory of the controller; in response to detecting that the current displacement of the controller is greater than a preset distance threshold, a first movement speed is determined based on the current displacement of the controller and the preset distance threshold, wherein, the current displacement is determined by the current position of the controller and the initial position of the controller, the initial position of the controller being the position of the controller when it begins to control the movement of the target object; controlling the target object to move based on the first movement speed, thereby realizing that when the user interacts with the extended reality scenario using the controller, the movement mode of the target object is determined based on the displacement of the controller, rather than based on the current touch position of the user's controller and the previous touch position. The control of objects in the extended reality scenario in the above interaction method conforms to the user's intuitive operating habits, thus improving the user's sense of control over objects in the extended reality scenario and improving the efficiency of the user's interaction with objects in the extended reality scenario. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of an interaction with an extended reality scene provided in an embodiment of this disclosure;

[0014] Figure 2 A flowchart illustrating an interactive method for extending real-world scenarios provided in this disclosure. Figure 1 ;

[0015] Figure 3 A flowchart illustrating an interactive method for extending real-world scenarios provided in this disclosure. Figure 2 ;

[0016] Figure 4 This is a schematic flowchart of the interactive method disclosed herein for extending real-world scenarios;

[0017] Figure 5 A structural block diagram of an interactive device for extending a real-world scene, provided as an embodiment of this disclosure;

[0018] Figure 6This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] In extended reality scenarios, users can interact with the environment. Through this interaction, users can control the movement of objects within the extended reality scene.

[0021] Please refer to Figure 1 This diagram illustrates an interaction within an extended reality scene. Users can experience an extended reality scene using the head-mounted extended reality device shown in Extended Reality device 101. Users can also interact with the extended reality scene using controller 102, etc.

[0022] In one example, a controller (e.g., a joystick) is set up to communicate with the extended reality device. The controller can have buttons for up, down, left, and right. When using the controller to move a target object in the extended reality scene, the user can operate these buttons on the joystick to move the target object within the extended reality scene.

[0023] In one example, a controller is set up to communicate with the extended reality device. This controller can include inertial sensors, etc. When the controller is used to move a target object in the extended reality scene, the processor of the extended reality device can bind the target object to the controller. The user can then perform movement operations on the controller to move the target object's position.

[0024] In these examples, the object the user needs to control is the controller. The controller has a touchpad or directional keys (the touchpad will be used as an example below). The user can perform touch operations on the touchpad or press the directional keys. The extended reality device can receive the current touch information sent by the controller and determine the user's control operation on the object in the extended reality scene based on the current touch information and the touch information at the previous moment. Since the object the user is operating is the controller, the direction of movement performed by the user on the controller may not be consistent with the direction of movement of the object in the extended reality scene implemented by the user through the controller. For example, to move the object to the left in the extended reality scene, one must first slide right on the touchpad and then slide left to control the object to move to the left. Therefore, controlling the movement of objects in the extended reality scene in this way does not conform to the user's intuitive operating habits, resulting in a weak sense of control over the objects in the extended reality scene. In addition, the efficiency of interacting with objects in the extended reality scene in this way is also not high.

[0025] To address the aforementioned issues, the solution provided in this disclosure controls the movement of the target object based on the positional offset between the controller's current position and its initial position when the user interacts with the extended reality scene using a controller. The initial position is the controller's position when it begins to move the target object during an interaction. This interaction method aligns with the user's intuitive operating habits, thus enhancing the user's sense of control over objects in the extended reality scene and improving the efficiency of interaction.

[0026] Please refer to Figure 2 , Figure 2 A flowchart illustrating an interactive method for extending real-world scenarios provided in this disclosure. Figure 1 The method includes:

[0027] S201: Based on the controller's motion control operation on the target object in the extended reality scene, control the target object to move based on the target movement trajectory; wherein, the target movement trajectory is determined by the controller's movement trajectory.

[0028] The execution entity of this disclosure embodiment can be an extended reality device or a server that establishes a communication connection with the extended reality device.

[0029] The extended reality scene here can be a virtual reality scene, an augmented reality scene, or a mixed reality scene.

[0030] The target object here can be a virtual object displayed in the extended reality scene described above. It can be any virtual object, such as a virtual object or a virtual character.

[0031] The controller here can be a physical electronic device that communicates with the extended reality device and is used to interact with the extended reality scene, such as a gamepad or joystick. Alternatively, the controller can also be the user's hand.

[0032] The aforementioned executing entity can establish a virtual three-dimensional space using the reference point of the aforementioned extended reality device as the origin coordinate. The extended reality scene can be displayed based on the aforementioned virtual three-dimensional space. In the aforementioned extended reality scene, the aforementioned controller can be displayed as a preset display object. This preset display object can be a virtual object, such as a virtual hand.

[0033] Users can operate the controller to change position, and the extended reality device can determine the user's interaction information to control objects in the extended reality scene based on the detected position change information of the controller.

[0034] In some application scenarios, the aforementioned extended reality device is equipped with an image acquisition device. The image acquisition device can acquire multiple images of the controller and determine the position of the controller relative to the extended reality device in the real scene based on the multiple images of the controller.

[0035] In other application scenarios, the controller can be equipped with sensors for determining its position. The controller can detect its own position using these sensors and then send the position information to the augmented reality device via the aforementioned communication connection. The augmented reality device then determines the position of the controller relative to itself in the real-world scene based on the sensor positions and its own position.

[0036] In some application scenarios, the position of the controller relative to the extended reality device in the real scene can be converted into the position of the virtual object corresponding to the controller in the three-dimensional space of the extended reality scene (the position of the virtual object of the controller in the three-dimensional space of the extended reality scene can be regarded as the position of the controller in the three-dimensional space of the extended reality scene).

[0037] As one implementation, the aforementioned execution entity can detect whether the virtual hand (the virtual object of the controller) is in contact with a target object in the extended reality scene. In practice, a collision detection algorithm (such as a bounding box-based collision detection algorithm) can be used to detect whether the virtual hand is in contact with the target object in the extended reality scene. After detecting that the virtual hand is in contact with the target object, and detecting that the contact duration of the virtual hand with the target object exceeds a preset duration threshold and / or that the virtual hand controls the target object to change position, it can be considered that the controller is controlling the movement of the target object.

[0038] The initial position of the controller can be the position of the controller when it starts controlling the movement of the target object.

[0039] When the controller moves the target object, if the controller's displacement is less than or equal to a preset distance threshold, the controller's movement trajectory can be determined. Then, based on the controller's movement trajectory, the target object's target movement trajectory is determined, and the target object is controlled to move along the target movement trajectory.

[0040] In some application scenarios, the movement trajectory of the controller in the real scene can be mapped to the extended reality scene, and the movement trajectory of the controller in the extended reality scene can be used as the target movement trajectory of the target object.

[0041] In some application scenarios, the controller's movement trajectory in an extended reality scenario can also be preset with geometric transformations to obtain the target movement trajectory.

[0042] In other words, when the displacement of the controller is less than or equal to a preset distance threshold, a small-scale displacement motion control method can be used to control the movement of the target object. Under the small-scale displacement motion control method, the movement trajectory of the target object can be determined based on the movement trajectory of the controller, and the target object can be controlled to move according to the target movement trajectory.

[0043] The displacement here is determined by the controller's current position and the controller's initial position as described above.

[0044] The current position here can be the position of the controller in the real-world scene as determined by the extended reality device at the current moment, based on the method described above for determining the controller's position. The controller's current position and initial position can respectively include the coordinates on the x, y, and z axes in the world coordinate system.

[0045] In some application scenarios, the current and initial positions of the controller in the real-world scene can be determined. Based on these positions, the current displacement in the real-world scene can be calculated. Then, the current displacement in the real-world scene is mapped to the extended reality scene using coordinate transformation to obtain the controller's current displacement in the extended reality scene.

[0046] Assuming the controller's initial position in the real-world scene is (x0, y0, z0) and its current position is (x1, y1, z1), the current displacement d in the real-world scene can be determined by the following formula:

[0047]

[0048] When the current displacement d is less than the preset distance threshold, the target object's movement can be controlled based on the target movement trajectory determined by the controller's movement trajectory.

[0049] A preset distance threshold for the real-world scenario can be set in advance. This preset distance threshold can be set according to the application scenario. In some application scenarios, the preset distance threshold can be determined by the range within which a person's arm can operate smoothly. Schematic, the preset distance threshold can be 30 centimeters. It is understood that, taking the controller's initial position in the real-world scenario as the origin, the controller can control the movement of the target object according to the method provided in this disclosure when moving in any direction.

[0050] The aforementioned execution entity can acquire the controller's position at preset time intervals. Then, based on the controller's position, it determines the controller's movement trajectory within the preset time interval, thereby determining the target object's target movement trajectory. The target object is then controlled to move along the target movement trajectory. The preset time interval can be set according to specific application scenarios and is not limited here. For example, 1 second could be used.

[0051] In some embodiments, step S201 includes controlling the target object to move based on a target movement trajectory and a second movement speed, wherein the second movement speed is determined by the movement speed of the controller.

[0052] The movement speed of the controller described above can be the movement speed of the controller in a real-world scenario. In these embodiments, the movement speed of the controller in an extended real-world scenario can be determined based on the mapping relationship between the movement speed in the real-world scenario and the movement speed in the extended real-world scenario.

[0053] As one implementation method, the motion speed of the controller in the extended reality scene can be used as the second motion speed of the target object.

[0054] In one implementation, the movement speed of the controller described above can be the movement speed of the controller in the real-world scene. In these implementations, a positive correlation coefficient can be set with the movement speed of the controller. The product of this positive correlation coefficient and the controller's movement speed in the real-world scene is used as the second movement speed of the target object in the extended real-world scene. That is, the second movement speed of the target object is positively correlated with the movement speed of the controller that controls its movement.

[0055] In these embodiments, when the controller's displacement is less than a preset distance threshold, the controller's movement trajectory and speed in the real-world scene can be acquired. This movement trajectory is then mapped to the extended reality scene to obtain the target movement trajectory. A second speed for the target object in the extended reality scene is then determined, and the target object is controlled to move according to the target movement trajectory and the second speed. Thus, in this small-scale displacement motion control mode, the target object's movement can be adjusted with fine granularity based on changes in the controller's motion, helping to improve the user's sense of control over objects in the extended reality scene and thereby improving the user's interaction efficiency with the extended reality scene.

[0056] S202: In response to detecting that the current displacement of the controller is greater than a preset distance threshold, a first motion speed is determined based on the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the target object to move.

[0057] S203: Control the target object to move based on the first motion speed.

[0058] To simplify the description, the controller's motion speed, displacement, and preset distance threshold are used as the motion speed, displacement, and preset distance threshold in the real-world scenario for explanation. The target object's first motion speed and second motion speed are used as the first motion speed and second motion speed in the extended real-world scenario for explanation.

[0059] Once the current displacement of the controller exceeds the preset distance threshold, the movement speed of the target object can be controlled using a large-scale displacement motion control method.

[0060] In large-scale displacement motion control, after determining the first motion speed, the aforementioned execution entity can configure the first motion speed for the aforementioned target object in the extended reality scenario, so that the target object moves according to the first motion speed.

[0061] In large-scale displacement motion control mode, the first motion speed can be determined by the displacement of the controller and the preset distance threshold, thereby controlling the target object to move at the first motion speed to achieve rapid movement of the target object.

[0062] In this embodiment, based on the controller's motion control operation on the target object in the extended reality scene, the target object is controlled to move based on a target movement trajectory; wherein, the target movement trajectory is determined by the controller's movement trajectory; in response to detecting that the controller's current displacement is greater than a preset distance threshold, a first movement speed is determined based on the controller's current displacement and the preset distance threshold, wherein, the current displacement is determined by the controller's current position and the controller's initial position, the controller's position when it begins to control the target object's movement; the target object is controlled to move based on the first movement speed. When the user interacts with the target object in the extended reality scene using the controller, the movement of the target object is controlled by the displacement determined by the controller's current position and initial position, rather than by the user's current touch position and historical touch positions on the controller. The above interaction method for controlling objects in the extended reality scene conforms to the user's intuitive operating habits and can improve the efficiency of user interaction with the extended reality scene.

[0063] Please refer to Figure 3 , Figure 3 A flowchart illustrating an interactive method for extending real-world scenarios provided in this disclosure. Figure 2 The method includes:

[0064] S301: Based on the controller's motion control operation on the target object in the extended reality scene, control the target object to move based on the target movement trajectory; wherein, the target movement trajectory is determined by the controller's movement trajectory.

[0065] The execution entity in this embodiment can be an extended reality device or a server that establishes a communication connection with the extended reality device.

[0066] The specific implementation of step S301 above can be related to Figure 2 Step S201 in the illustrated embodiment is the same or similar, and will not be described in detail here.

[0067] S302: In response to detecting that the current displacement of the controller is greater than a preset distance threshold, determine the difference between the current displacement and the preset distance threshold, and determine the first motion speed based on the difference; wherein, the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object.

[0068] S303: Control the target object to move based on the first motion speed.

[0069] When the current displacement of the controller is greater than the preset distance threshold, the first motion speed can be determined based on the current displacement and the preset distance threshold.

[0070] In this embodiment, the controller's motion speed, displacement, and preset distance threshold are the motion speed, displacement, and preset distance threshold in the real scene, and the target object's first motion speed and second motion speed are described using the first motion speed and second motion speed in the extended real scene.

[0071] Specifically, the difference between the current displacement and the preset distance threshold can be calculated. The formula for calculating the difference w is as follows:

[0072] w = dR (2);

[0073] Where d is the current displacement in the real scene as determined by formula (1). R is the preset distance threshold in the real scene.

[0074] The first velocity is determined based on the difference mentioned above.

[0075] As an illustrative example, the first velocity can be determined by multiplying the aforementioned difference by a preset coefficient. This coefficient can be a constant or it can vary with the difference.

[0076] The aforementioned difference gradually increases from zero as the controller moves, thus the aforementioned first motion speed also gradually increases.

[0077] In some embodiments, step S302 includes: determining the difference between the current displacement and a preset distance threshold, and determining a first motion speed according to a preset exponential mapping function of the difference.

[0078] In these embodiments, the first velocity described above can be determined by the following formula (3):

[0079] v2(t)=(P1(t)-R) n ×SG (3);

[0080] Where v2(t) is the first velocity of the target object at time t; P1(t) is the displacement of the controller at time t; R is a preset distance threshold; (P1(t)-R) is the difference between the current displacement of the controller and the preset distance threshold; n is a value greater than 0; SG is the velocity gain, with a value greater than or equal to zero. SG is a preset constant that can be set according to the specific application scenario.

[0081] In these embodiments, under large-scale displacement motion control, the first motion velocity of the target object is determined using a preset exponential mapping function of the above-mentioned difference.

[0082] In small-scale displacement motion control mode, the second motion speed of the target object is determined by the motion speed of the controller. In large-scale displacement motion control mode, the first motion speed is determined by a preset exponential mapping function of the difference between the current displacement and a preset distance threshold. During the transition from small-scale to large-scale displacement motion control mode, the aforementioned difference gradually increases from small to large. The first motion speed determined by the exponential mapping function of the aforementioned difference does not produce a sudden change compared to the second motion speed, thus achieving a smooth transition between the first and second motion speeds determined by the motion speed of the controller. This prevents visually abrupt speed changes when transitioning from small-scale to large-scale displacement motion control mode.

[0083] In some embodiments, step 303 includes the following sub-steps:

[0084] First, determine the initial position of the target object at the previous moment.

[0085] Secondly, based on the displacement obtained by integrating the first position and the first velocity over time within the current time period, the second position of the target object at the current moment is determined, where the current time period is the time period between the current moment and the previous moment.

[0086] Finally, the target object is controlled to move to the second position based on the first motion speed within the current time period.

[0087] In these embodiments, the target object moves according to the target trajectory determined by the controller's movement trajectory under small-scale displacement motion control. The moment when the controller's movement distance equals a preset distance threshold is designated as the reference starting time t0. The target object can then be controlled to move according to large-scale displacement motion control, using this reference starting time as the time starting point.

[0088] After the reference start time, the displacement of the controller can be sampled once according to the preset sampling frequency, and the first motion velocity of the target object can be determined based on the displacement of the controller.

[0089] The duration of the current time period mentioned above can be set according to the specific application scenario, and there is no restriction here. In some scenarios, the duration of the current time period can be 1 second. That is, the time difference between the current moment and the previous moment is 1 second.

[0090] Specifically, the second position of the target object can be determined by the following formula (4):

[0091]

[0092] Among them, P2(t) i-1 The position of the target object at the previous moment is considered the first position. P2(t) i) is the position to be moved to by the target object at the current moment, which is regarded as the second position; P1(t) is the displacement of the controller over time during the current time period; R is the preset distance threshold; n is a value greater than 0; SG is the preset speed gain, and the magnitude of the preset speed gain can be set according to the specific application scenario; i is an integer greater than or equal to 1.

[0093] It is understandable that the second position of the target object at the current moment will be regarded as the first position in the next moment.

[0094] For the first time interval [t0, t1] after the reference start time, the position of the target object at the reference start time t0 can be taken as the first position p2(t0) of the reference start time. The first velocity of the target object within the first time interval is determined according to formula (3) above. The second position at time t1 is determined according to formula (4) above. Based on the second position of the target object at time t1, the target object is controlled to move to the second position at the first velocity.

[0095] For the second time interval [t1, t2] after the reference start time, the target object's position at time t1 can be taken as the target object's first position p2(t1). The first velocity of the target object within the second time interval is determined according to formula (3) above. The second position of the target object is determined according to formula (4) above. The target object is controlled to move to the aforementioned second position according to the first velocity within the second time interval.

[0096] In these embodiments, under the large-scale displacement motion control mode, the target object is controlled to move according to the first motion speed and the second position in time intervals, thereby realizing the rapid and accurate movement of the target object under the control of the controller in the large-scale displacement motion control mode.

[0097] In this embodiment, when the controller displacement is small, the target object moves according to the controller's trajectory and a second speed determined by the controller's speed under small-scale displacement motion control. The target object's motion can be controlled with fine granularity by the controller. When the controller displacement is large, the target object moves according to a first speed determined by the current displacement and a preset distance threshold, and a second position determined based on the first speed, under large-scale displacement motion control. Furthermore, this method balances the motion control of the target object under both small-scale and large-scale displacement motion control methods, ensuring consistent interaction logic. Therefore, the transition from small-scale to large-scale displacement motion control avoids visual discontinuity. This improves the user's sense of control over objects in the extended reality scene, thereby enhancing the interaction efficiency between the user and objects in the extended reality scene.

[0098] In some embodiments, the above-described interactive method for extended reality scenes further includes: displaying a virtual 3D controller in the extended reality scene.

[0099] In these embodiments, if the current displacement exceeds a preset distance threshold, a virtual 3D controller can be displayed in the extended reality scene. By displaying the virtual 3D controller in the extended reality scene, the user can be alerted that the displacement of the currently operated controller has exceeded the preset distance threshold.

[0100] In some application scenarios, the controller's initial position is its position in the real-world scene. The above-mentioned methods for displaying a virtual 3D controller in an extended reality scene include:

[0101] The virtual 3D controller is displayed at its initial position, which is the mapped position in the extended reality scene.

[0102] In these application scenarios, the virtual 3D controller is displayed at the mapped position of the target object in the extended reality scene at the initial position in the real scene. This can show the starting position of the controller in the extended reality scene for this interactive operation, and also prompt the user that the movement of the target object in this interactive operation has entered a large-scale range.

[0103] exist Figure 2 and Figure 3 In some embodiments of the interactive method for extending real-world scenarios provided in the illustrated embodiments, the controller may include one of the following:

[0104] Hand, manual controller, and eye-tracking controller.

[0105] In some applications, the controller described above can be a human hand. When the controller is a human hand, the extended reality device can determine the position of the hand by capturing images of the hand, and then control the movement of the target object by controlling the changes in the position of the hand.

[0106] In some application scenarios, the aforementioned controller can be a manual controller, which can be any type of controller operated by hand, such as a handle.

[0107] In these application scenarios, augmented reality devices can determine the position of the hand controller by acquiring images of the hand controller, and the hand controller can determine its position based on the position detected by its own position detection sensors. Then, motion control can be performed on target objects in the augmented reality scene based on changes in the hand controller's position.

[0108] In some applications, the controller described above can also be various eye-tracking controllers. These eye-tracking controllers can be equipped with sensors to determine eye position, such as infrared sensors and cameras, to capture the user's eye position. The augmented reality device can receive the eye position data sent by the eye-tracking controller and then control the motion of the target object based on changes in eye position.

[0109] In these embodiments, the physical controller may not have buttons. When the user controls the target object in the extended reality scene through the controller, the extended reality device realizes motion control of the target object in the extended reality scene based on the displacement generated between the current position and the initial position of the controller. This allows the user to experience the same feeling as when the user directly operates the target object in the real scene, which can improve the user experience.

[0110] Please refer to Figure 4 The diagram illustrates a principle flowchart of an extended reality scene interaction method. An image acquisition device can be installed on the extended reality device to capture images of indicator lights on the controller. The executing entity can locate the controller using the light spots from the indicator lights in the images, thereby determining the controller's position.

[0111] Once the aforementioned execution entity detects that the controller's location is within an extended reality scene, it can display the virtual object corresponding to the controller within the extended reality scene. This virtual object could be, for example, a virtual hand.

[0112] The aforementioned execution entity can obtain the position of the controller when the target object begins to move, and use that position as the initial position.

[0113] The user can then move the controller, and the aforementioned execution entity can control the movement of the target object by detecting the relationship between the controller's current position and its initial position.

[0114] When the displacement of the user-controlled movement controller is detected to be no greater than a preset distance threshold, that is, when the controller is moved within the preset distance threshold, the target object in the extended reality scenario moves along with the controller. Specifically, the target object's movement can be controlled based on the controller's movement trajectory, the target movement trajectory determined by the controller's movement speed, and a second movement speed.

[0115] When the user's displacement relative to the controller is detected to be greater than a preset distance threshold, the difference between the displacement and the preset distance is determined, and a first motion velocity related to this difference is determined. A second position is determined based on the first position of the target object at the previous moment and the displacement determined by the integral of the first motion velocity over time within the current time period. The target object is then controlled to move to the second position based on the first motion velocity.

[0116] Corresponding to the interactive method for extending real-world scenarios in the above embodiments, Figure 5 This is a structural block diagram of an interactive device for extending real-world scenarios, provided as an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 5 The device 50 includes: a first control unit 501, a determining unit 502, and a second control unit 503. Wherein:

[0117] The first control unit 501 is used to control the target object to move based on the target movement trajectory according to the motion control operation of the controller on the target object in the extended reality scene; wherein the target movement trajectory is determined by the movement trajectory of the controller.

[0118] The determining unit 502 is used to determine a first motion speed based on the current displacement of the controller and the preset distance threshold in response to detecting that the current displacement of the controller is greater than a preset distance threshold. The current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the target object to move.

[0119] The second control unit 503 is used to control the target object to move based on the first motion speed.

[0120] In some embodiments of this disclosure, the first control unit 501 is further configured to:

[0121] The target object is controlled to move based on the target movement trajectory and a second motion speed, wherein the second motion speed is determined by the motion speed of the controller.

[0122] In some embodiments of this disclosure, the determining unit 502 is further configured to:

[0123] Determine the difference between the current displacement and the preset distance threshold, and determine the first motion velocity based on the difference.

[0124] In some embodiments of this disclosure, the determining unit 502 is further configured to:

[0125] The first velocity is determined based on a preset exponential mapping function of the difference.

[0126] In some embodiments of this disclosure, the second control unit 503 is further configured to:

[0127] Determine the first position of the target object at the previous moment;

[0128] The second position of the target object at the current moment is determined by integrating the first position and the first velocity over time within the current time period. The current time period is the time period between the current moment and the previous moment.

[0129] Controlling the target object to move to a second position based on a first motion speed within the current time period. In some embodiments of this disclosure, device 50 further includes a display unit (not shown in the figures), the display unit being used for:

[0130] When controlling the movement of the target object at a first motion speed, a virtual 3D controller is displayed in an extended reality scene.

[0131] In some embodiments of this disclosure, the initial position is a position in the real scene, and the display unit is further configured to: display a virtual 3D controller at the initial position at a mapped position in the extended real scene.

[0132] In some embodiments of this disclosure, the controller includes one of the following:

[0133] Hand, manual controller, or eye-tracking controller.

[0134] To implement the above embodiments, this disclosure also provides an electronic device.

[0135] refer to Figure 6 The diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The electronic device 600 can be an extended reality device or a server. The extended reality device can include various wearable devices, such as head-mounted extended reality devices. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0136] like Figure 6 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0137] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0138] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0139] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0140] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0141] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0142] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not necessarily limiting in certain circumstances; for example, the first control unit can also be described as "a unit that controls the movement of a target object based on a target trajectory according to the motion control operations of the controller on the target object in the extended reality scene".

[0145] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0146] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] In a first aspect, according to one or more embodiments of this disclosure, an interactive method for extending real-world scenarios is provided, comprising:

[0148] Based on the controller's motion control operations on the target object in the extended reality scene, the target object is controlled to move based on the target movement trajectory; wherein, the target movement trajectory is determined by the controller's movement trajectory;

[0149] In response to detecting that the current displacement of the controller is greater than a preset distance threshold, a first motion speed is determined based on the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object;

[0150] The target object is controlled to move based on a first motion speed.

[0151] According to one or more embodiments of this disclosure, controlling the target object to move based on a target trajectory according to the controller's motion control operation on the target object in an extended reality scene includes:

[0152] The target object is controlled to move based on the target movement trajectory and a second motion speed, wherein the second motion speed is determined by the motion speed of the controller.

[0153] According to one or more embodiments of this disclosure, determining a first motion speed based on the current displacement of the controller and a preset distance threshold includes:

[0154] Determine the difference between the current displacement and the preset distance threshold, and determine the first motion velocity based on the difference.

[0155] According to one or more embodiments of this disclosure, determining a first motion speed based on a difference includes:

[0156] The first velocity is determined based on a preset exponential mapping function of the difference.

[0157] According to one or more embodiments of this disclosure, controlling a target object to move based on a first motion speed includes:

[0158] Determine the first position of the target object at the previous moment;

[0159] The second position of the target object at the current moment is determined by integrating the first position and the first velocity over time within the current time period. The current time period is the time period between the current moment and the previous moment.

[0160] Control the target object to move to the second position based on the first movement speed within the current time period.

[0161] According to one or more embodiments of this disclosure, the method further includes:

[0162] When controlling the movement of the target object at a first motion speed, a virtual 3D controller is displayed in an extended reality scene.

[0163] According to one or more embodiments of this disclosure, the initial position is a position in a real-world scene, and a virtual 3D controller is displayed in an extended reality scene, including:

[0164] The virtual 3D controller is displayed at its initial position, which is the mapped position in the extended reality scene.

[0165] According to one or more embodiments of this disclosure, the controller includes one of the following:

[0166] Hand, manual controller, or eye-tracking controller.

[0167] Secondly, according to one or more embodiments of this disclosure, an interactive device for extending real-world scenarios is provided, comprising:

[0168] The first control unit is used to control the target object to move based on the target movement trajectory according to the motion control operation of the controller on the target object in the extended reality scene; wherein the target movement trajectory is determined by the movement trajectory of the controller.

[0169] The determining unit is used to determine a first motion speed based on the current displacement of the controller and the preset distance threshold in response to detecting that the current displacement of the controller is greater than a preset distance threshold. The current displacement is determined by the current position of the controller and the initial position of the controller, where the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object.

[0170] The second control unit is used to control the target object to move based on the first motion speed.

[0171] According to one or more embodiments of this disclosure, the first control unit is further configured to:

[0172] The target object is controlled to move based on the target movement trajectory and a second motion speed, wherein the second motion speed is determined by the motion speed of the controller.

[0173] According to one or more embodiments of this disclosure, the determining unit is further configured to:

[0174] Determine the difference between the current displacement and the preset distance threshold, and determine the first motion velocity based on the difference.

[0175] According to one or more embodiments of this disclosure, the determining unit is further configured to:

[0176] The first velocity is determined based on a preset exponential mapping function of the difference.

[0177] According to one or more embodiments of this disclosure, the second control unit is further configured to:

[0178] Determine the first position of the target object at the previous moment;

[0179] The second position of the target object at the current moment is determined by integrating the first position and the first velocity over time within the current time period. The current time period is the time period between the current moment and the previous moment.

[0180] Controlling the target object to move to a second position based on a first motion speed within the current time period. According to one or more embodiments of this disclosure, the device further includes a display unit, the display unit being used for:

[0181] When controlling the movement of the target object at a first motion speed, a virtual 3D controller is displayed in an extended reality scene.

[0182] According to one or more embodiments of this disclosure, the initial position is a position in a real scene, and the display unit is further configured to: display a virtual 3D controller at a mapped position in an extended real scene at the initial position.

[0183] According to one or more embodiments of this disclosure, the controller includes one of the following:

[0184] Hand, manual controller, or eye-tracking controller.

[0185] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0186] The memory stores instructions that the computer executes;

[0187] At least one processor executes computer execution instructions stored in memory, causing at least one processor to perform the first aspect above and various possible interactive methods for extending the real-world scenario involved in the first aspect.

[0188] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, implement the first aspect above and various possible interactive methods for extending real-world scenarios involved in the first aspect.

[0189] Fifthly, according to one or more embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the first aspect above and various possible interactive methods for extending real-world scenarios involved in the first aspect.

[0190] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0191] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0192] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An interactive method for extending real-world scenarios, comprising: Based on the controller's motion control operations on the target object in the extended reality scene, the target object is controlled to move based on a target movement trajectory; wherein, the target movement trajectory is determined by the controller's movement trajectory; In response to detecting that the current displacement of the controller is greater than a preset distance threshold, the difference between the current displacement of the controller and the preset distance threshold is determined, and a first motion speed is determined based on the difference. The current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the spatial position in the real scene when the controller starts to control the movement of the target object. Control the target object to move based on the first motion speed; It also includes: if the current displacement of the controller is less than or equal to a preset distance threshold, then the target movement trajectory of the target object is determined based on the movement trajectory of the controller.

2. The method of claim 1, wherein, The step of controlling the target object to move based on a target trajectory according to the controller's motion control operation on the target object in the extended reality scene includes: In response to detecting that the current displacement of the controller is less than or equal to a preset distance threshold, the target object is controlled to move based on a target movement trajectory and a second movement speed, wherein the second movement speed is determined by the movement speed of the controller.

3. The method according to claim 1, characterized in that, Determining the first motion speed based on the difference includes: The first motion speed is determined based on a preset exponential mapping function of the difference.

4. The method according to claim 1, characterized in that, The control of the target object to move based on the first motion speed includes: Determine the first position of the target object at the previous moment; Based on the displacement obtained by integrating the first position and the first movement speed over time in the current time period, the second position of the target object at the current moment is determined, wherein the current time period is the time period between the current moment and the previous moment. Control the target object to move to the second position based on the first movement speed within the current time period.

5. The method according to claim 1, characterized in that, The method further includes: When the target object is controlled to move at the first motion speed, a virtual 3D controller is displayed in an extended reality scene.

6. The method according to claim 5, characterized in that, The initial position is the position in the real scene, and the display of the virtual 3D controller in the extended reality scene includes: The virtual 3D controller is displayed at its initial position, which is the mapped position in the extended reality scene.

7. The method according to any one of claims 1 to 6, characterized in that, The controller includes one of the following: Hand, manual controller, or eye-tracking controller.

8. An interactive device for extending a real-world scene, comprising: The first control unit is configured to control the target object to move based on a target movement trajectory according to the motion control operation of the controller on the target object in the extended reality scene; wherein the target movement trajectory is determined by the movement trajectory of the controller; A determining unit is configured to, in response to detecting that the current displacement of the controller is greater than a preset distance threshold, determine the difference between the current displacement of the controller and the preset distance threshold, and determine a first motion velocity based on the difference, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, the initial position of the controller being the spatial position in the real scene when the controller begins to control the movement of the target object; if the current displacement of the controller is less than or equal to the preset distance threshold, then the target movement trajectory of the target object is determined based on the movement trajectory of the controller. The second control unit is used to control the target object to move based on the first motion speed.

9. An electronic device, comprising: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the interactive method for extending a real-world scene as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the interactive method for extending a real-world scenario as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the interactive method for extending a real-world scene as described in any one of claims 1 to 7.

Images