Interaction method and device for augmented reality scene and electronic equipment

Abstract

The embodiment of the invention provides an interaction method and device for an augmented reality scene and electronic equipment, and the method comprises the steps: controlling a target object to move based on a target movement track according to the motion control operation of a controller on the target object in the augmented reality scene; wherein the target moving track is determined by the moving track of the controller; in response to the fact that the detected current displacement of the controller is larger than a preset distance threshold value, the first movement speed is determined according to the current displacement of the controller and the preset distance threshold value, and the current displacement is determined by the current position of the controller and the initial position of the controller; the initial position of the controller is the position of the controller when the controller starts to control the target object to move; and controlling the target object to move based on the first movement speed. Control on the object in the augmented reality scene in the interaction mode conforms to the operation habit of the user, and the interaction efficiency of the user and the object in the augmented reality scene is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more particularly to an interactive method, device, and electronic device for extending a reality scene. Background Art

[0002] Due to the characteristics of extended reality technology such as immersion, interactivity and imagination, applications based on extended reality are increasingly appearing in different industries, such as gaming and entertainment, home decoration and other industries.

[0003] In an extended reality scene, users can interact with objects in the extended reality scene from a long distance, such as performing operations such as moving the position of objects that appear in the extended reality scene.

[0004] The inventors have discovered that the current method of operating objects in an extended reality scene is not intuitive, resulting in low efficiency in user interaction with objects in the extended reality scene. Summary of the invention

[0005] The embodiments of the present disclosure provide an interactive method, device and electronic device for an extended reality scene, so as to overcome the problem of low interaction efficiency between a user and objects in the extended reality scene due to the unintuitive operation mode of objects in the extended reality scene in the related art.

[0006] In a first aspect, an embodiment of the present disclosure provides an interactive method for an extended reality scene, the method comprising: controlling the target object in the extended reality scene to move based on a target movement trajectory according to a motion control operation of the controller on the target object; wherein the target movement trajectory is determined by the movement trajectory of the controller; in response to detecting that a current displacement of the controller is greater than a preset distance threshold, determining a first movement speed according to the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object; controlling the target object to move based on the first movement speed.

[0007] In a second aspect, an embodiment of the present disclosure provides an interactive device for an extended reality scene, the device comprising: a first control unit, for controlling the target object in the extended reality scene to move based on a target movement trajectory according to a motion control operation of the controller on the target object; wherein the target movement trajectory is determined by the movement trajectory of the controller; a determination unit, for determining a first movement speed according to the current displacement of the controller and the preset distance threshold in response to detecting that the current displacement of the controller is greater than a preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object; and a second control unit, for controlling the target object to move based on the first movement speed.

[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the first aspect above and various possible interactive methods for extending the reality scene involved in the first aspect.

[0009] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the above-mentioned first aspect and various possible interactive methods for extending the reality scene involved in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the first aspect as well as various possible interactive methods for extending reality scenarios as described above.

[0011] The interactive method, device and electronic device for an extended reality scene provided in this embodiment control the target object to move based on a target movement trajectory according to the motion control operation of the controller on the target object in the extended reality scene; wherein the target movement trajectory is determined by the movement trajectory of the controller; in response to detecting that the current displacement of the controller is greater than a preset distance threshold, a first movement speed is determined according to the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object; the target object is controlled to move based on the first movement speed, so that when the user uses the controller to interact with the extended reality scene, the movement mode of the target object is determined according to the displacement of the controller, rather than controlling the movement of the target object in the extended reality scene according to the current touch position and the previous touch position of the user's controller. The control of the objects in the extended reality scene in the above-mentioned interactive mode conforms to the intuitive operation habits of the user, so it can improve the user's sense of control over the objects in the extended reality scene and improve the efficiency of the user's interaction with the objects in the extended reality scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 A schematic diagram of a scene for interaction with an extended reality scene provided by an embodiment of the present disclosure;

[0014] Figure 2 A flow chart of an interactive method for augmenting a reality scene provided by an embodiment of the present disclosure Figure 1 ;

[0015] Figure 3 A flow chart of an interactive method for augmenting a reality scene provided by an embodiment of the present disclosure Figure 2 ;

[0016] Figure 4 A schematic flow chart of the interactive method for extending the reality scene disclosed in the present invention;

[0017] Figure 5 A structural block diagram of an interactive device for augmenting a reality scene provided by an embodiment of the present disclosure;

[0018] Figure 6A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0020] In the extended reality scene, the user can interact with the extended reality scene. Through the interaction, the user can control the movement of objects in the extended reality scene.

[0021] Please refer to Figure 1 , which shows a schematic diagram of an interaction in an extended reality scene. A user can experience an extended reality scene using a head-mounted extended reality device shown as an extended reality device 101. A user can also interact with the extended reality scene using a controller 102 or the like.

[0022] In one example, a controller (e.g., a joystick) is provided to communicate with the extended reality device. The controller may be provided with up, down, left, and right buttons. When the controller is used to control the movement of a target object in the extended reality scene, the user may operate the buttons on the joystick to move the target object in the extended reality scene.

[0023] In one example, a controller is provided to achieve communication connection with the extended reality device, and an inertial sensor or the like may be provided in the controller. When the controller is used to control the movement of a target object in the extended reality scene, the processor of the extended reality device may bind the target object to the controller. The user may perform a movement operation on the controller to move the position of the target object.

[0024] In these examples, the object that the user needs to control is the controller. The controller has a touchpad or a direction key (the touchpad is used as an example below). The user can perform touch operations on the above touchpad, or press operations on the direction keys, etc. The extended reality device can receive the current touch information sent by the controller, and determine the user's control operation on the object in the extended reality scene based on the current touch information and the touch information at the previous moment. Since the object operated by the user is the controller, the direction of movement performed by the user on the controller may not be consistent with the direction of movement of the object in the extended reality scene implemented by the user through the controller. For example, the above object needs to be moved to the left in the extended reality scene, but it is necessary to slide to the right on the touchpad first, and then slide to the left to control the left movement of the above object. Therefore, controlling the movement of objects in the extended reality scene in the above manner does not conform to the user's intuitive operating habits, resulting in a weak sense of control over the objects in the extended reality scene. In addition, the efficiency of interacting with objects in the extended reality scene in the above manner is not high.

[0025] In order to improve the above problems, the solution provided by the present disclosure controls the movement of the target object according to the position offset between the position of the controller and the initial position of the controller when the user uses the controller to interact with the extended reality scene. The initial position is the position of the controller when the controller starts to control the movement of the target object in an interactive operation. The control of objects in the extended reality scene in the above interaction method conforms to the user's intuitive operating habits, so it can improve the user's sense of control over the objects in the extended reality scene, and can also improve the efficiency of interacting with objects in the extended reality scene.

[0026] Please refer to Figure 2 , Figure 2 A flow chart of an interactive method for augmenting a reality scene provided by an embodiment of the present disclosure Figure 1 The method comprises:

[0027] S201: According to the motion control operation of the controller on the target object in the extended reality scene, the target object is controlled to move based on a target movement trajectory; wherein the target movement trajectory is determined by the movement trajectory of the controller.

[0028] The execution subject of the embodiment of the present disclosure may be an extended reality device, or may be a server that realizes communication connection with the extended reality device.

[0029] The extended reality scene here can be a virtual reality scene, an augmented reality scene, or a mixed reality scene.

[0030] The target object here may be a virtual object displayed in the above-mentioned augmented reality scene, and may be any virtual object, such as a virtual object, a virtual character, etc.

[0031] The controller here can be an actual electronic device that is connected to the extended reality device for communication and is used to interact with the extended reality scene, such as a handle, a joystick, etc. In addition, the controller can also be a user's hand, etc.

[0032] The execution subject may establish a virtual three-dimensional space using the reference point of the extended reality device as the origin coordinate. The extended reality scene may be displayed based on the virtual three-dimensional space. In the extended reality scene, the controller may be displayed as a preset display object. The preset display object may be a virtual object, such as a virtual hand.

[0033] The user can operate the controller to change the position, and the extended reality device can determine the interaction information of the user controlling the object in the extended reality scene based on the detected position change information of the controller.

[0034] In some application scenarios, the above-mentioned extended reality device is provided with an image collector, which can collect multiple images of the controller and determine the position of the controller relative to the extended reality device in the real scene based on the multiple images of the controller.

[0035] In some other application scenarios, a sensor for determining a position may be provided on the controller, and the controller may detect its own position through the sensor, and then send the position information to the extended reality device through the communication connection. The extended reality device determines the position of the controller relative to the extended reality device in the real scene based on the position of the sensor and its own position.

[0036] In some application scenarios, the position of the controller in the real scene relative to the extended reality device can be converted into the position of the virtual object corresponding to the controller in the three-dimensional space of the extended reality scene (the position of the virtual object of the above controller in the three-dimensional space of the extended reality scene can be regarded as the position of the controller in the three-dimensional space of the extended reality scene).

[0037] As an implementation method, the above-mentioned execution subject can detect whether the virtual hand (the virtual object of the controller) is in contact with the target object in the extended reality scene. In practice, a collision detection algorithm (such as a collision detection algorithm based on a bounding box) can be used to detect whether the virtual hand is in contact with the target object in the extended reality scene. After the virtual hand is detected to be in contact with the above-mentioned target object, and it is detected that the virtual hand's contact time with the position of the above-mentioned target object is greater than a preset time threshold and / or the virtual hand controls the target object to change position, it can be regarded as the controller controlling the movement of the target object.

[0038] The position of the controller when the controller starts to control the movement of the target object can be used as the initial position of the controller.

[0039] When the controller controls the target object to move, if the displacement of the controller is less than or equal to a preset distance threshold, the movement trajectory of the controller when controlling the target object to move can be determined. Then, the target movement trajectory of the target object is determined according to the movement trajectory of the controller, and the target object is controlled to move along the target movement trajectory.

[0040] In some application scenarios, the movement trajectory of the controller in the real scene can be mapped to the extended reality scene, and the movement trajectory of the controller in the extended reality scene is used as the target movement trajectory of the target object.

[0041] In some application scenarios, a preset geometric transformation may be performed on the movement trajectory of the controller in the augmented reality scene to obtain a target movement trajectory.

[0042] That is, when the displacement of the controller is less than or equal to the preset distance threshold, the small-scale displacement motion control method can be used to control the motion of the target object. In the small-scale displacement motion control method, the motion trajectory of the target object can be determined according to the motion trajectory of the controller, and the target object can be controlled to move according to the target motion trajectory.

[0043] The displacement here is determined by the current position of the controller and the initial position of the controller mentioned above.

[0044] The current position here can be the position of the controller in the real scene determined by the augmented reality device at the current moment according to the above method of determining the position of the controller. The current position and initial position of the controller can respectively include the coordinates of the x, y, and z axes in the world coordinate system.

[0045] In some application scenarios, the current position and initial position of the controller in the real scene can be determined, and the current displacement in the real scene can be calculated based on the current position and initial position. Then, the current displacement in the real scene is mapped to the extended reality scene according to the coordinate transformation to obtain the current displacement of the controller in the extended reality scene.

[0046] Assuming that the initial position of the controller in the real scene is (x0, y0, z0) and the current position is (x1, y1, z1), the current displacement d in the real scene can be determined by the following formula:

[0047]

[0048] When the current displacement magnitude d is smaller than a preset distance threshold, the movement of the target object may be controlled according to the target movement trajectory determined by the movement trajectory of the controller.

[0049] The preset distance threshold in the real scene can be preset, and the preset distance threshold here can be set according to the application scenario. In some application scenarios, the above-mentioned preset distance threshold can be determined by the range that a person's arm can operate smoothly. Schematically, the above-mentioned preset distance threshold can be 30 centimeters. It can be understood that, with the initial position of the controller in the real scene as the origin, the controller can control the movement of the target object according to the method provided by the present disclosure when moving in any direction.

[0050] The execution subject may obtain the position of the controller once every preset time period. Then, the movement trajectory of the controller within the time period is determined according to the position of the controller, and then the target movement trajectory of the target object is determined. Then, the target object is controlled to move along the target movement trajectory. The preset time period may be set according to the specific application scenario, and is not limited here. Schematically, the preset time period may be, for example, 1 second.

[0051] In some embodiments, the above step S201 includes: controlling the target object to move based on the target movement trajectory and the second movement speed, wherein the second movement speed is determined by the movement speed of the controller.

[0052] The movement speed of the above-mentioned controller may be the movement speed of the controller in the real scene. In these embodiments, the movement speed of the controller in the extended reality scene may be determined based on the mapping relationship between the movement speed in the real scene and the movement speed in the extended reality scene.

[0053] As an implementation manner, the movement speed of the controller in the augmented reality scene may be used as the second movement speed of the target object.

[0054] As an implementation, the movement speed of the controller may be the movement speed of the controller in the real scene. In these implementations, a positive correlation coefficient with the movement speed of the controller may be set, and the product of the positive correlation coefficient and the movement speed of the controller in the real scene is used as the second movement speed of the target object in the extended real scene. That is, the second movement speed of the target object is positively correlated with the movement speed of the controller that controls its movement.

[0055] In these embodiments, when the displacement of the controller is less than a preset distance threshold, the movement trajectory and movement speed of the controller in the real scene can be obtained, the movement trajectory is mapped to the extended reality scene to obtain the target movement trajectory, and then the second movement speed of the target object in the extended reality scene is determined, and the target object is controlled to move according to the target movement trajectory and the second movement speed. Therefore, under the small-scale displacement motion control mode, the movement of the target object can achieve fine-grained position adjustment according to the movement change of the controller, which helps to improve the user's control sense of objects in the extended reality scene, thereby improving the interaction efficiency between the user and the extended reality scene.

[0056] S202: In response to detecting that a current displacement of the controller is greater than a preset distance threshold, determining a first movement speed according to the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by a current position of the controller and an initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object.

[0057] S203: Control the target object to move based on the first movement speed.

[0058] To simplify the description, the controller's movement speed, displacement and preset distance threshold are used as the movement speed, displacement and preset distance threshold in the real scene for explanation below, and the first movement speed and second movement speed of the target object are used as the first movement speed and second movement speed in the extended reality scene for explanation.

[0059] After the current displacement of the controller is greater than the preset distance threshold, the movement speed of the target object can be controlled by using a large-scale displacement motion control method.

[0060] In the large-scale displacement motion control method, after the first motion speed is determined, the execution subject can configure the first motion speed for the target object in the extended reality scene so that the target object moves at the first motion speed.

[0061] In the large-scale displacement motion control mode, the first motion speed can be jointly determined according to the displacement of the controller and the preset distance threshold, so as to control the target object to move at the first motion speed to achieve rapid movement of the target object.

[0062] In this embodiment, according to the motion control operation of the controller on the target object in the extended reality scene, the target object is controlled to move based on the target movement trajectory; wherein the target movement trajectory is determined by the movement trajectory of the controller; in response to detecting that the current displacement of the controller is greater than a preset distance threshold, a first movement speed is determined according to the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object; the target object is controlled to move based on the first movement speed. When the user uses the controller to interact with the target object in the extended reality scene, the movement of the target object is controlled by the displacement determined by the current position and the initial position of the controller, rather than controlling the movement of the target object in the extended reality scene according to the current touch position and the historical touch position of the controller by the user. The control of objects in the extended reality scene in the above interaction method conforms to the user's intuitive operating habits and can improve the efficiency of the user's interaction with the extended reality scene.

[0063] Please refer to Figure 3 , Figure 3 A flow chart of an interactive method for augmenting a reality scene provided by an embodiment of the present disclosure Figure 2 The method comprises:

[0064] S301: According to the motion control operation of the controller on the target object in the extended reality scene, the target object is controlled to move based on a target movement trajectory; wherein the target movement trajectory is determined by the movement trajectory of the controller.

[0065] The execution subject of this embodiment may be an extended reality device, or may be a server that is in communication with the extended reality device.

[0066] The specific implementation of the above step S301 can be Figure 2 Step S201 in the illustrated embodiment is the same or similar and will not be described in detail here.

[0067] S302: In response to detecting that the current displacement of the controller is greater than a preset distance threshold, determining the difference between the current displacement and the preset distance threshold, and determining a first movement speed based on the difference; wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object.

[0068] S303: Control the target object to move based on a first movement speed.

[0069] When the current displacement of the controller is greater than a preset distance threshold, the first movement speed may be determined according to the current displacement and the preset distance threshold.

[0070] In this embodiment, the movement speed, displacement and preset distance threshold of the controller are the movement speed, displacement and preset distance threshold in the real scene, and the first movement speed and the second movement speed of the target object are described with the first movement speed and the second movement speed in the extended reality scene.

[0071] Specifically, the difference between the current displacement and the preset distance threshold may be calculated. The calculation formula for the above difference w is as follows:

[0072] w = dR (2);

[0073] Wherein, d is the current displacement size in the real scene determined by formula (1), and R is the preset distance threshold in the real scene.

[0074] The first movement speed is determined according to the above difference.

[0075] As an illustrative example, the difference may be multiplied by a preset coefficient to determine the first movement speed. The coefficient may be a constant or may vary with the difference.

[0076] The difference gradually increases from zero as the controller moves, so that the first movement speed also gradually increases.

[0077] In some embodiments, the above step S302 includes: determining a difference between the current displacement and a preset distance threshold, and determining the first movement speed according to a preset exponential mapping function of the difference.

[0078] In these embodiments, the first movement speed may be determined by the following formula (3):

[0079] v2(t)=(P1(t)-R) n ×SG (3);

[0080] Wherein, v2(t) is the first motion speed of the target object at the current time t; P1(t) is the displacement of the controller at the current time t; R is the preset distance threshold; (P1(t)-R) is the difference between the current displacement of the controller and the preset distance threshold; n is a value greater than 0; SG is the speed gain, which is greater than or equal to zero. SG is a preset constant that can be set according to the specific application scenario.

[0081] In these embodiments, in the large-scale displacement motion control mode, the preset exponential mapping function of the difference is used to determine the first motion speed of the target object.

[0082] In the small-scale displacement motion control mode, the second motion speed of the target object is determined by the motion speed of the controller, and in the large-scale displacement motion control mode, the first motion speed is determined by a preset exponential mapping function of the difference between the current displacement and the preset distance threshold. In the process of converting the small-scale displacement motion control mode to the large-scale displacement motion control mode, the above difference gradually increases from small to large, and the first motion speed determined by the exponential mapping function of the above difference will not produce a sudden change compared to the second motion speed, so that a smooth transition can be achieved between the first motion speed and the second motion speed determined by the motion speed of the controller, so that when the small-scale displacement motion control mode is converted to the large-scale displacement motion control mode, there will be no visual phenomenon of a sudden change in speed.

[0083] In some embodiments, step 303 includes the following sub-steps:

[0084] First, the first position of the target object at the previous moment is determined.

[0085] Secondly, the second position of the target object corresponding to the current moment is determined according to the displacement obtained by integrating the first position and the first movement speed over time in the current time period, where the current time period is the time period between the current moment and the previous moment.

[0086] Finally, the target object is controlled to move to a second position based on the first movement speed within the current time period.

[0087] In these embodiments, the target object moves according to the target movement trajectory determined by the movement trajectory of the controller in the small-scale displacement motion control mode. The moment when the movement distance of the controller is equal to the preset distance threshold is the reference starting moment t0. The reference starting moment can be used as the time starting point to control the target object to move according to the large-scale displacement motion control mode.

[0088] After the reference start time, the displacement of the controller may be collected once according to a preset sampling frequency, and a first movement speed of the target object may be determined once according to the displacement of the controller.

[0089] The duration of the current time period can be set according to the specific application scenario and is not limited here. In some scenarios, the duration of the current time period can be 1 second. That is, the time difference between the current moment and the previous moment is 1 second.

[0090] Specifically, the second position of the target object can be determined by the following formula (4):

[0091]

[0092] Among them, P2(t i-1 ) is the position of the target object at the previous moment, which is regarded as the first position. i) is the position to which the target object is to be moved at the current moment, which is regarded as the second position; P1(t) is the displacement of the controller that changes with time in the current time period; R is the preset distance threshold; n is a value greater than 0, SG is the preset speed gain, and the size of the preset speed gain can be set according to the specific application scenario; i is an integer greater than or equal to 1.

[0093] It can be understood that the second position of the target object at the current moment will be regarded as the first position at the next moment.

[0094] For the first time period [t0, t1] after the reference start time, the position of the target object at the reference start time t0 can be taken as the first position p2(t0) at the reference start time. The first movement speed of the target object in the first time period is determined according to the above formula (3). The second position at time t1 is determined according to the above formula (4). Based on the second position of the target object at time t1, the target object is controlled to move to the second position at the first movement speed.

[0095] For the second time period [t1, t2] after the reference start time, the position of the target object at time t1 can be taken as the first position p2(t1) of the target object. The first movement speed of the target object in the second time period is determined according to the above formula (3). The second position of the target object is determined according to the above formula (4). The target object is controlled to move to the above second position according to the first movement speed in the second time period.

[0096] In these embodiments, under the large-scale displacement motion control mode, the target object is controlled to move according to the first motion speed and the second position in each time period, thereby achieving fast and accurate movement of the target object according to the control of the controller under the large-scale displacement motion control mode.

[0097] In this embodiment, when the displacement of the controller is small, the target object moves according to the movement trajectory of the controller and the second movement speed determined by the movement speed of the controller under the small-scale displacement motion control mode, and the movement of the target object can be fine-grained controlled by the controller. When the displacement of the controller is large, the target object moves according to the first movement speed determined by the current displacement and the preset distance threshold and the second position determined based on the first movement speed under the large-scale displacement motion control mode. In addition, the above method takes into account the motion control of the target object under the small-scale displacement motion control mode and the large-scale displacement motion control mode, and the interactive logic of the motion control of the target object tends to be consistent. Therefore, the visual discontinuity is avoided by entering the large-scale displacement motion control mode from the small-scale displacement motion control mode. The user's sense of control over the objects in the extended reality scene is improved, which is conducive to improving the interaction efficiency between the user and the objects in the extended reality scene.

[0098] In some embodiments, the above-mentioned interaction method for an extended reality scene also includes: displaying a virtual three-dimensional controller in the extended reality scene.

[0099] In these embodiments, if the current displacement is greater than the preset distance threshold, a virtual 3D controller may be displayed in the extended reality scene. By displaying the virtual 3D controller in the extended reality scene, the user may be prompted that the displacement of the controller currently being operated has exceeded the preset distance threshold.

[0100] In some application scenarios, the initial position of the controller is the position in the real scene, and the above-mentioned displaying the virtual three-dimensional controller in the extended reality scene includes:

[0101] The virtual three-dimensional controller is displayed at the mapped position in the extended reality scene at the initial position.

[0102] In these application scenarios, the above-mentioned virtual three-dimensional controller is displayed at the mapping position of the extended reality scene at the initial position in the real scene. On the one hand, it can display the starting position of the controller of the target object in this interactive operation in the extended reality scene, and can also prompt the user that the movement of the target object in this interactive operation has entered a large-scale range.

[0103] exist Figure 2 and Figure 3 In some embodiments of the interactive method for augmenting the reality scene provided by the illustrated embodiment, the controller may include one of the following:

[0104] Human hands, hand controllers, and eye controllers.

[0105] In some application scenarios, the controller may be a human hand. When the controller is a human hand, the extended reality device may determine the position of the human hand by collecting an image of the human hand, and then control the movement of the target object by the change in the position of the human hand.

[0106] In some application scenarios, the controller may be a manual controller, where the manual controller may be any manual controller, such as a handle.

[0107] In these application scenarios, the extended reality device can determine the position of the controller by collecting images of the human hand controller, and the human hand controller can determine the position of the human hand controller based on the position detected by its own sensor for position detection. Then, the target object in the extended reality scene can be controlled in motion according to the position change of the human hand controller.

[0108] In some application scenarios, the controller may also be various eye movement controllers. The eye movement controller may be provided with a sensor for determining the eye position, such as an infrared sensor and a camera to capture the user's eye position. The extended reality device may receive the eye position sent by the eye movement controller, and then control the movement of the target object according to the change of the eye position.

[0109] In these embodiments, buttons may not be set in the above-mentioned physical controller. When the user controls the target object in the extended reality scene through the controller, the extended reality device realizes motion control of the target object in the extended reality scene according to the displacement between the current position of the controller and the initial position of the controller, so that the user can experience the same feeling as the user directly operating the target object in the real scene, which can improve the user experience.

[0110] Please refer to Figure 4 , which shows a principle flow chart of the extended reality scene interaction method. An image acquisition device can be set on the extended reality device, and the image acquisition device can acquire an image of the indicator light on the controller. The execution subject can locate the controller by the light spot of the indicator light in the image, thereby determining the position of the controller.

[0111] After detecting that the position of the controller is in the extended reality scene, the execution subject may display a virtual object corresponding to the controller in the extended reality scene. The virtual object may be, for example, a virtual hand.

[0112] The execution subject may obtain the position of the controller when the controller controls the target object to start moving, and use the position as the initial position.

[0113] The user may subsequently move the controller, and the execution subject may control the movement of the target object by detecting the relationship between the current position and the initial position of the controller.

[0114] When it is detected that the displacement of the user's moving controller is not greater than the preset distance threshold, that is, when the controller is moved within the preset distance threshold, the target object is controlled to move with the movement of the controller in the extended reality scene. Specifically, the target object can be controlled to move according to the movement trajectory of the controller and the target movement trajectory determined by the movement speed of the controller and the second movement speed.

[0115] When it is detected that the displacement of the controller by the user is greater than a preset distance threshold, the difference between the displacement and the preset distance is determined, and a first movement speed related to the difference is determined. The second position is determined according to the first position of the target object at the previous moment and the displacement determined by the integral of the first movement speed over time in the current time period, and the target object is controlled to move to the second position based on the first movement speed.

[0116] Corresponding to the interactive method for extending the reality scene in the above embodiment, Figure 5 This is a block diagram of the structure of the interactive device for augmenting the reality scene provided by the embodiment of the present disclosure. For the convenience of explanation, only the part related to the embodiment of the present disclosure is shown. Figure 5 , the device 50 includes: a first control unit 501, a determination unit 502, and a second control unit 503. Wherein:

[0117] A first control unit 501, configured to control the target object to move based on a target movement trajectory according to a motion control operation of the controller on the target object in the extended reality scene; wherein the target movement trajectory is determined by a movement trajectory of the controller;

[0118] a determining unit 502, configured to determine a first motion speed according to the current displacement of the controller and the preset distance threshold in response to detecting that the current displacement of the controller is greater than a preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object;

[0119] The second control unit 503 is used to control the target object to move based on the first movement speed.

[0120] In some embodiments of the present disclosure, the first control unit 501 is further configured to:

[0121] The target object is controlled to move based on the target movement trajectory and the second movement speed, wherein the second movement speed is determined by the movement speed of the controller.

[0122] In some embodiments of the present disclosure, the determining unit 502 is further configured to:

[0123] The difference between the current displacement and the preset distance threshold is determined, and the first movement speed is determined according to the difference.

[0124] In some embodiments of the present disclosure, the determining unit 502 is further configured to:

[0125] The first movement speed is determined according to a preset exponential mapping function of the difference.

[0126] In some embodiments of the present disclosure, the second control unit 503 is further configured to:

[0127] Determine a first position of the target object at a previous moment;

[0128] Determine a second position of the target object corresponding to the current moment according to a displacement obtained by integrating the first position and the first motion speed over time in a current time period, where the current time period is a time period between the current moment and a previous moment;

[0129] Control the target object to move to the second position based on the first movement speed within the current time period. In some embodiments of the present disclosure, the device 50 further includes a display unit (not shown in the figure), and the display unit is used to:

[0130] When controlling the target object to move at the first movement speed, displaying a virtual three-dimensional controller in the extended reality scene.

[0131] In some embodiments of the present disclosure, the initial position is a position in the real scene, and the display unit is further used to: display the virtual three-dimensional controller at the mapping position of the initial position in the extended reality scene.

[0132] In some embodiments of the present disclosure, the controller includes one of the following:

[0133] Human hand, hand controller or eye controller.

[0134] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0135] refer to Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure, and the electronic device 600 may be an extended reality device or a server. The extended reality device may include various wearable devices, such as a head-mounted extended reality device, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0136] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0137] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0138] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0139] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0140] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0141] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0142] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the unit does not limit the unit itself in some cases. For example, the first control unit may also be described as "a unit that controls the movement of the target object based on the target movement trajectory according to the motion control operation of the controller on the target object in the extended reality scene."

[0145] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] In a first aspect, according to one or more embodiments of the present disclosure, there is provided an interactive method for augmenting a reality scene, comprising:

[0148] According to the motion control operation of the controller on the target object in the extended reality scene, the target object is controlled to move based on the target movement trajectory; wherein the target movement trajectory is determined by the movement trajectory of the controller;

[0149] In response to detecting that the current displacement of the controller is greater than a preset distance threshold, determining a first movement speed according to the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object;

[0150] The target object is controlled to move based on a first movement speed.

[0151] According to one or more embodiments of the present disclosure, according to the motion control operation of the controller on the target object in the extended reality scene, controlling the target object to move based on the target movement trajectory includes:

[0152] The target object is controlled to move based on the target movement trajectory and the second movement speed, wherein the second movement speed is determined by the movement speed of the controller.

[0153] According to one or more embodiments of the present disclosure, determining a first motion speed according to a current displacement of a controller and a preset distance threshold includes:

[0154] The difference between the current displacement and the preset distance threshold is determined, and the first movement speed is determined according to the difference.

[0155] According to one or more embodiments of the present disclosure, determining the first motion speed according to the difference includes:

[0156] The first movement speed is determined according to a preset exponential mapping function of the difference.

[0157] According to one or more embodiments of the present disclosure, controlling the target object to move based on a first movement speed includes:

[0158] Determine a first position of the target object at a previous moment;

[0159] Determine a second position of the target object corresponding to the current moment according to a displacement obtained by integrating the first position and the first motion speed over time in a current time period, where the current time period is a time period between the current moment and a previous moment;

[0160] The target object is controlled to move to a second position based on the first movement speed within a current time period.

[0161] According to one or more embodiments of the present disclosure, the method further includes:

[0162] When controlling the target object to move at the first movement speed, displaying a virtual three-dimensional controller in the extended reality scene.

[0163] According to one or more embodiments of the present disclosure, the initial position is a position in a real scene, and displaying a virtual three-dimensional controller in an extended reality scene includes:

[0164] The virtual three-dimensional controller is displayed at the mapped position in the extended reality scene at the initial position.

[0165] According to one or more embodiments of the present disclosure, the controller includes one of the following:

[0166] Human hand, hand controller or eye controller.

[0167] In a second aspect, according to one or more embodiments of the present disclosure, there is provided an interactive device for augmenting a reality scene, comprising:

[0168] A first control unit, configured to control the target object to move based on a target movement trajectory according to a motion control operation of the controller on the target object in the extended reality scene; wherein the target movement trajectory is determined by a movement trajectory of the controller;

[0169] a determining unit, configured to determine a first motion speed according to the current displacement of the controller and the preset distance threshold in response to detecting that the current displacement of the controller is greater than a preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object;

[0170] The second control unit is used to control the target object to move based on the first movement speed.

[0171] According to one or more embodiments of the present disclosure, the first control unit is further configured to:

[0172] The target object is controlled to move based on the target movement trajectory and the second movement speed, wherein the second movement speed is determined by the movement speed of the controller.

[0173] According to one or more embodiments of the present disclosure, the determining unit is further configured to:

[0174] The difference between the current displacement and the preset distance threshold is determined, and the first movement speed is determined according to the difference.

[0175] According to one or more embodiments of the present disclosure, the determining unit is further configured to:

[0176] The first movement speed is determined according to a preset exponential mapping function of the difference.

[0177] According to one or more embodiments of the present disclosure, the second control unit is further configured to:

[0178] Determine a first position of the target object at a previous moment;

[0179] Determine a second position of the target object corresponding to the current moment according to a displacement obtained by integrating the first position and the first motion speed over time in a current time period, where the current time period is a time period between the current moment and a previous moment;

[0180] Control the target object to move to the second position based on the first movement speed within the current time period. According to one or more embodiments of the present disclosure, the device further includes a display unit, and the display unit is used to:

[0181] When controlling the target object to move at the first movement speed, displaying a virtual three-dimensional controller in the extended reality scene.

[0182] According to one or more embodiments of the present disclosure, the initial position is a position in the real scene, and the display unit is further used to: display the virtual three-dimensional controller at the mapping position of the initial position in the extended reality scene.

[0183] According to one or more embodiments of the present disclosure, the controller includes one of the following:

[0184] Human hand, hand controller or eye controller.

[0185] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0186] Memory stores computer-executable instructions;

[0187] At least one processor executes computer-executable instructions stored in the memory, so that the at least one processor executes the first aspect as well as various possible interactive methods for extending the reality scene as described above.

[0188] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the first aspect and various possible interactive methods for extending the reality scene as described above are implemented.

[0189] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the first aspect as well as various possible interactive methods for extending reality scenes as described above in the first aspect.

[0190] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0191] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0192] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An interactive method for extending a reality scene, comprising: According to the motion control operation of the controller on the target object in the extended reality scene, the target object is controlled to move based on the target movement trajectory; wherein the target movement trajectory is determined by the movement trajectory of the controller; In response to detecting that the current displacement of the controller is greater than a preset distance threshold, determining a first movement speed according to the current displacement of the controller and the preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object; The target object is controlled to move based on the first movement speed.

2. The method according to claim 1, characterized in that According to the motion control operation of the controller on the target object in the extended reality scene, controlling the target object to move based on the target movement trajectory includes: The target object is controlled to move based on a target movement trajectory and a second movement speed, wherein the second movement speed is determined by a movement speed of the controller.

3. The method according to claim 1, characterized in that The step of determining the first movement speed according to the current displacement of the controller and a preset distance threshold comprises: A difference between the current displacement and a preset distance threshold is determined, and a first movement speed is determined according to the difference.

4. The method according to claim 3, characterized in that: The determining the first movement speed according to the difference comprises: The first movement speed is determined according to a preset exponential mapping function of the difference.

5. The method according to claim 1, characterized in that The controlling the target object to move based on the first movement speed includes: Determine a first position of the target object at a previous moment; Determine a second position of the target object corresponding to a current moment according to a displacement obtained by integrating the first position and the first movement speed over time in a current time period, wherein the current time period is a time period between the current moment and the previous moment; The target object is controlled to move to a second position based on the first movement speed within a current time period.

6. The method according to claim 1, characterized in that The method further comprises: When controlling the target object to move at the first movement speed, a virtual three-dimensional controller is displayed in the extended reality scene.

7. The method according to claim 6, characterized in that The initial position is a position in a real scene, and displaying a virtual three-dimensional controller in an extended reality scene includes: The virtual three-dimensional controller is displayed at the mapped position in the extended reality scene at the initial position.

8. The method according to any one of claims 1 to 7, characterized in that The controller includes one of the following: Human hand, hand controller or eye controller.

9. An interactive device for augmenting a reality scene, comprising: A first control unit, configured to control the target object in the extended reality scene to move based on a target movement trajectory according to a motion control operation of the controller on the target object; wherein the target movement trajectory is determined by a movement trajectory of the controller; a determining unit, configured to determine a first motion speed according to the current displacement of the controller and the preset distance threshold in response to detecting that the current displacement of the controller is greater than a preset distance threshold, wherein the current displacement is determined by the current position of the controller and the initial position of the controller, and the initial position of the controller is the position of the controller when the controller starts to control the movement of the target object; A second control unit is used to control the target object to move based on the first movement speed.

10. An electronic device, comprising: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the interactive method for extending the reality scene as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the interactive method for extending the reality scene as described in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the interactive method for extending the reality scene as described in any one of claims 1 to 8 is implemented.

Images