Interaction method of an ar device and mobile device

Abstract

The application provides an interaction method of an AR device and a mobile device, which are used for providing multiple interaction modes and improving user experience. The method comprises the following steps: determining a target mode based on a current pose of the mobile device; if the target mode is a handle mode, acquiring the current pose of the mobile device every first specified time length, controlling the pose of a screen cursor in the AR device based on the current pose, and completing an interaction operation by using a target determination instruction to control the screen cursor of the AR device when the target determination instruction is received; if the target mode is a touch mode, projecting by using a micro projector in the mobile device to obtain a target recording area and a non-recording area; if an interaction instruction is an input operation of a user in the recording area, determining a target sliding track in response to the input operation, and superimposedly displaying the target sliding track in a display interface in the AR device; and if the interaction instruction is an interaction instruction of the user in the non-recording area, controlling the display interface in the AR device based on the interaction instruction.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and more particularly to an interaction method for an AR device and a mobile device. Background Technology

[0002] As an emerging wearable device, the design of the interaction method for AR (Augmented Reality) glasses is crucial to the user experience. Currently, the most common interaction method is through a handheld controller. The handheld controller used with AR glasses, similar to a game controller, can provide more precise and complex operations.

[0003] However, existing handheld controllers only allow users to interact with AR glasses by moving the corresponding controller in their hands, which is relatively simple and results in a poor user experience. Summary of the Invention

[0004] This application provides an interaction method and mobile device for AR devices, which can provide multiple ways to interact with AR devices and improve user experience.

[0005] In a first aspect, embodiments of this application provide an interaction method for an object-oriented AR device, the method comprising:

[0006] In response to an interactive command sent by the user, a target mode is determined based on the current pose of the mobile device;

[0007] If the target mode is the controller mode, the current pose of the mobile device is obtained every first specified time interval, the pose of the screen cursor in the AR device is controlled based on the current pose of the mobile device, and when a target determination instruction is received from the user, the screen cursor in the AR device is controlled to complete the interactive operation using the target determination instruction.

[0008] If the target mode is a touch mode, then a target recording area and a non-recording area are obtained by projecting through the micro projector in the mobile device; if the interaction command is an input operation by the user in the recording area, then in response to the input operation, the target sliding trajectory is determined and the target sliding trajectory is superimposed on the display interface in the AR device; if the interaction command is an interaction command by the user in the non-recording area, then the display interface in the AR device is controlled based on the interaction command.

[0009] A second aspect of this application provides a mobile device including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0010] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0011] In response to an interactive command sent by the user, a target mode is determined based on the current pose of the mobile device;

[0012] If the target mode is the controller mode, the current pose of the mobile device is obtained every first specified time interval, the pose of the screen cursor in the AR device is controlled based on the current pose of the mobile device, and when a target determination instruction is received from the user, the screen cursor in the AR device is controlled to complete the interactive operation using the target determination instruction.

[0013] If the target mode is a touch mode, then a target recording area and a non-recording area are obtained by projecting through the micro projector in the mobile device; if the interaction command is an input operation by the user in the recording area, then in response to the input operation, the target sliding trajectory is determined and the target sliding trajectory is superimposed on the display interface in the AR device; if the interaction command is an interaction command by the user in the non-recording area, then the display interface in the AR device is controlled based on the interaction command.

[0014] According to a third aspect of the present invention, a computer storage medium is provided, the computer storage medium storing a computer program for performing the method as described in the first aspect.

[0015] In the above embodiments of this application, a target mode is determined based on the current pose of the mobile device in response to an interaction command sent by the user. If the target mode is a controller mode, the current pose of the mobile device is acquired every first specified time interval, and the pose of the screen cursor in the AR device is controlled based on the current pose of the mobile device. When a target determination command is received from the user, the screen cursor in the AR device is controlled using the target determination command to complete the interaction operation. If the target mode is a touch mode, a target recording area and a non-recording area are obtained by projecting through a micro projector in the mobile device. If the interaction command is an input operation by the user in the recording area, a target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed on the display interface in the AR device. If the interaction command is an interaction command by the user in the non-recording area, the display interface in the AR device is controlled based on the interaction command. Thus, the embodiments of this application provide multiple interaction modes, expand the ways of AR interaction, and improve the user experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 An exemplary diagram illustrating an application scenario provided in an embodiment of this application is shown;

[0018] Figure 2 An exemplary schematic diagram of the structure of a mobile device provided in an embodiment of this application is shown;

[0019] Figure 3 An exemplary flowchart of one of the interaction methods of the AR device provided in this application is shown;

[0020] Figure 4 An exemplary schematic diagram of the process for determining a target pattern provided in an embodiment of this application is shown;

[0021] Figure 5 An exemplary diagram of the controller mode provided in this application embodiment is shown;

[0022] Figure 6 An exemplary schematic diagram of the touch mode provided in an embodiment of this application is shown;

[0023] Figure 7A An exemplary diagram of the recording area provided in an embodiment of this application is shown;

[0024] Figure 7B An exemplary illustration shows a schematic diagram of an overlay display provided in an embodiment of this application;

[0025] Figure 8 An exemplary illustration shows a flowchart for determining a target sliding trajectory provided in an embodiment of this application;

[0026] Figure 9 An exemplary diagram illustrates the calibration in the projection area provided in an embodiment of this application;

[0027] Figure 10 An exemplary schematic diagram illustrates a process for determining the radius of a finger-touched area according to an embodiment of this application;

[0028] Figure 11 The second schematic flowchart of the interaction method of the AR device provided in the embodiment of this application is illustrated by example;

[0029] Figure 12 An exemplary structural diagram of the interactive device of the AR device provided in an embodiment of this application is shown. Detailed Implementation

[0030] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0031] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0032] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0033] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to be omnipresent but not exclusive; for example, a product or device comprising a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device.

[0034] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0035] The following is an overview of the ideas behind the embodiments of this application.

[0036] Current handheld controllers only allow users to interact with AR glasses by moving the corresponding controller in their hand. This limited functionality results in a poor user experience.

[0037] To address the issue of poor user experience due to limited interactive functions in existing technologies, this application provides an interaction method for AR devices. Responding to user-sent interaction commands, a target mode is determined based on the current pose of the mobile device. If the target mode is a controller mode, the current pose of the mobile device is acquired every first specified time interval, and the pose of the screen cursor in the AR device is controlled based on this pose. Upon receiving a target determination command from the user, the screen cursor in the AR device is controlled using this command to complete the interaction. If the target mode is a touch mode, a target recording area and a non-recording area are projected using a micro-projector in the mobile device. If the interaction command is a user input operation in the recording area, a target sliding trajectory is determined in response to the input operation, and this trajectory is overlaid on the display interface of the AR device. If the interaction command is a user interaction command in the non-recording area, the display interface of the AR device is controlled based on this command. Therefore, this application provides multiple interaction modes, expanding the ways of AR interaction and improving the user experience.

[0038] like Figure 1 As shown, an application scenario of an interaction method for an AR device is presented, which includes a mobile device 101 and an AR device 102.

[0039] In one possible application scenario, the mobile device 101 responds to an interaction command sent by the user and determines a target mode based on its current pose. If the target mode is a controller mode, the mobile device 101 acquires its current pose every first specified time interval, controls the pose of the screen cursor in the AR device 102 based on the current pose, and, upon receiving a target determination command from the user, controls the screen cursor in the AR device 102 to complete the interaction operation using the target determination command. If the target mode is a touch mode, the mobile device 101 projects a target recording area and a non-recording area using a micro-projector in the mobile device 101. If the interaction command is an input operation by the user in the recording area, the target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed on the display interface in the AR device 102. If the interaction command is an interaction command by the user in the non-recording area, the display interface in the AR device 102 is controlled based on the interaction command.

[0040] in, Figure 1 The mobile device 101 and the AR device 102 can interact with each other through a communication network. The communication network can be either wireless or wired.

[0041] For example, mobile device 101 can access the network via cellular mobile communication technology to communicate with AR device 102, wherein the cellular mobile communication technology includes, for example, 5th Generation Mobile Networks (5G) technology.

[0042] Optionally, the mobile device 101 can access the network and communicate with the AR device 102 via short-range wireless communication, wherein the short-range wireless communication method includes, for example, Wireless Fidelity (Wi-Fi) technology.

[0043] The description in this application focuses only on a single mobile device 101 and a single AR device 102. However, those skilled in the art should understand that the illustrated mobile device 101 and single AR device 102 are intended to illustrate the operation of the mobile device 101 and AR device 102 involved in the technical solutions of this application, and do not imply any limitation on the number, type, or location of the mobile device 101 and AR device 102. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application.

[0044] It should be noted that the AR device interaction method proposed in this application is not only applicable to... Figure 1 The application scenarios shown can also be applied to any interactive device with AR capabilities.

[0045] The interaction method of the AR device in the exemplary embodiment of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the above application scenarios are only shown for the purpose of facilitating the understanding of the methods and principles of this application, and the implementation of this application is not limited in any way in this respect.

[0046] Before introducing the interaction method of the AR device in this application, the mobile device in the embodiments of this application will be introduced first, such as... Figure 2 The diagram shown is a structural schematic of the mobile device in an embodiment of this application. Figure 2 As can be seen from the above, the mobile device 200 in this embodiment includes a micro projector 201, a touch panel 202, a first sensor 203, a second sensor 204, a system indicator light 205, a processor 206, a fixture 207, a memory 208, an IMU sensor 209, and a Bluetooth module 210. Wherein:

[0047] The memory 208 stores a computer program, and the processor 206 is configured to perform the following operations based on the computer program: in response to an interaction command sent by the user through the touch panel 202, determine a target mode based on the current pose of the mobile device acquired by the IMU sensor 209; if the target mode is a gamepad mode, then every first specified time interval, acquire the current pose of the mobile device, and send the current pose of the mobile device to the AR device through the Bluetooth module 210, so as to control the pose of the screen cursor in the AR device based on the current pose of the mobile device, and when a target determination command is received from the user, use the target determination command to control the screen cursor in the AR device to complete the interactive operation; if the target mode is a gamepad mode, then the processor 206 is configured to perform the following operations based on the target mode: in response to an interaction command sent by the user through the touch panel 202, determine a target mode based on the current pose of the mobile device ... If the target mode is touch mode, then the micro projector 201 in the mobile device projects the target recording area and non-recording area. If the interaction command is an input operation by the user in the recording area, then in response to the input operation, the first distance between the first sensor and the user's finger is obtained through the first sensor 203, and the second distance between the second sensor and the user's finger is obtained through the second sensor 204. The target sliding trajectory is determined based on the first distance and the second distance, and the target sliding trajectory is superimposed and displayed on the display interface of the AR device. If the interaction command is an interaction command by the user in the non-recording area, then the display interface of the AR device is controlled based on the interaction command.

[0048] The system indicator light 205 is used to send a notification signal to the user, and the fixture 207 is used to fix the mobile device.

[0049] The interaction method of the AR device in the embodiments of this application will be described below with reference to the accompanying drawings, such as... Figure 3 The diagram shown illustrates the interaction method for AR devices, which may include the following steps:

[0050] Step 301: In response to the user's interactive command, determine the target mode based on the current pose of the mobile device;

[0051] The method for determining the target pattern in the embodiments of this application will be described below. For example... Figure 4 The diagram shown illustrates the process for determining the target pattern, which may include the following steps:

[0052] Step 401: Every second specified time interval, obtain the current pose of the mobile device;

[0053] The second specified duration in this application embodiment can be set according to the actual situation, but this application embodiment does not limit the specific value of the second specified duration.

[0054] Step 402: Compare the current pose with the previous pose of the mobile device to determine if they are the same. If they are not the same, proceed to step 403. If they are the same, proceed to step 404.

[0055] Step 403: Determine that the target mode of the mobile device is the controller mode;

[0056] Step 404: Determine whether the number of times the current pose of the mobile device is obtained is greater than a specified number. If not, return to step 401; if yes, proceed to step 405.

[0057] Step 405: Determine the target model of the mobile device as the touch mode.

[0058] It should be noted that the specified quantity in this embodiment is 10, but the specific value of the specified quantity in this embodiment is not limited. The specific value of the specified quantity in this embodiment can be set according to the actual situation.

[0059] Step 302: If the target mode is the controller mode, then every first specified time interval, the current pose of the mobile device is obtained, and the pose of the screen cursor in the AR device is controlled based on the current pose of the mobile device.

[0060] like Figure 5 As shown, in the controller mode of this embodiment, the mobile device can be worn on the wrist of either of the user's arms. When the user swings their arm up, down, left, and right, the mobile device acquires its current pose and transmits it to the AR device via Bluetooth. The AR device receives the constantly updated current pose of the mobile device as the arm moves, determines the pose of the screen cursor in the AR device based on the current pose of the mobile device, and uses the determined screen cursor pose to control the pose of the screen cursor in the AR device.

[0061] In one embodiment, controlling the pose of the screen cursor in the AR device based on the current pose of the mobile device can be specifically implemented as follows:

[0062] By utilizing the pre-set correspondence between the pose of the mobile device and the pose of the screen cursor, a target pose of the screen cursor corresponding to the current pose of the mobile device is obtained based on the current pose of the mobile device, and the pose of the screen cursor in the AR device is controlled to move to the target pose.

[0063] Step 303: When a target determination instruction sent by the user is received, the target determination instruction is used to control the screen cursor in the AR device to complete the interactive operation;

[0064] In this embodiment, the purpose of swinging the arm to control the direction of the screen cursor in the AR device is to adjust the cursor's pointing direction. When the screen cursor points to the desired target, it needs to be confirmed by clicking. This confirmation can be performed using the other hand, which is not wearing the mobile device. Specifically, any finger of that hand touches the touch panel of the mobile device. If the touch duration exceeds 0.1 seconds, the mobile device's system recognizes the valid touch, and the system indicator light flashes blue. The mobile device then transmits the touch signal to the AR device via Bluetooth. Upon receiving the touch signal, the AR device recognizes it as a confirmation signal, thus completing the confirmation operation of the screen cursor pointing to the target and completing the interactive operation.

[0065] Step 304: If the target mode is a touch mode, then the target recording area and non-recording area are obtained by projecting through the micro projector in the mobile device;

[0066] like Figure 6 The diagram illustrates the touch mode. In this embodiment, the mobile device can be placed upright on any horizontal tabletop in touch mode, projecting a projection area onto the tabletop. The projection area in this embodiment maintains the same resolution as the AR device, and the area size is magnified by a specified factor based on the AR device's screen size. This specified factor is 4 times, but not limited to a single factor. Within this projection area, the user can write text by sliding their finger across the tabletop. The written text is then rendered overlay on the video, game, or application screen by the AR glasses, used to record information needed by the user during use.

[0067] In one embodiment, step 304 may be specifically implemented as follows: projecting through a micro projector in the mobile device to obtain a projection area; in response to the user's selection operation of preset recording areas in the projection area to obtain the target recording area; and determining the other projection areas in the projection area other than the target recording area as the non-recording areas.

[0068] The micro projector in this embodiment projects a specified number of preset recording areas into the projection area. For example... Figure 7A As shown, Figure 7A The white area in the image represents the preset recording area. Users can select one or more recording areas by swiping or tapping to define the target recording area. The selected target recording area then turns gray. The AR device's screen will then overlay this selected area onto the surface of the video, game, or application. The specific overlay display method is as follows: Figure 7B As shown. If the user finds the target recording area inconvenient and wants to change it, they simply need to click the selected target recording area again; the recording area will then be canceled and turn yellow again. If the user does not touch the projection area for more than 1 minute, the target recording area selection phase ends by default, and the target recording area in the AR device is determined. After the target recording area is determined, the user-selected target recording area in the projection area turns white (target recording area), and unselected recording areas turn black (non-recording area).

[0069] Step 305: If the interaction instruction is an input operation by the user in the recording area, then in response to the input operation, the target sliding trajectory is determined and the target sliding trajectory is superimposed and displayed on the display interface of the AR device;

[0070] The method for determining the target sliding trajectory in step 305 is described below. For example... Figure 8 The diagram shown illustrates the process of determining the target sliding trajectory, which may include the following steps:

[0071] Step 801: When the user's finger begins to touch the recording area, the first sensor of the AR handle acquires a specified number of first distances, wherein the first distance is the distance between the first sensor and the user's finger;

[0072] In this embodiment of the application, the user's finger may begin to touch the recording area for the first time, or the user may touch the recording area, remove their finger, and then touch the recording area again.

[0073] If a user touches the recording area for the first time, the user can place their finger on a marker point in the projection area, specifically as follows: Figure 9 As shown, the user places the object for approximately 1 second each time, and the number of placements is 10. The first and second sensors collect the first and second distances respectively. The system indicator light turns green after each successful collection and turns red if the collection fails. The system indicator light turns off after the collection is completed.

[0074] Step 802: Obtain a specified number of second distances using the second sensor of the AR controller, wherein the second distance is the distance between the second sensor and the user's finger;

[0075] It should be noted that the execution order of steps 801 and 802 is not limited in this embodiment. Step 801 can be executed first and then step 802; step 802 can be executed first and then step 801; or steps 801 and 802 can be executed simultaneously.

[0076] Step 803: Based on the specified number of first distances and the specified number of second distances, obtain a specified number of distance sets, wherein any distance set includes the first distance and the second distance;

[0077] In the embodiments of this application, the first distance and the second distance in any distance set are obtained at the same time.

[0078] Step 804: Based on a specified number of distance sets, obtain the radius of the finger-touched area;

[0079] like Figure 10 The diagram shown illustrates the process for determining the radius of the area touched by a finger, which may include the following steps:

[0080] Step 1001: For any set of distances, obtain the center of the finger touch area corresponding to the set of distances based on the first distance and the second distance in the set of distances; wherein, the abscissa of the center of the finger touch area corresponding to the set of distances can be obtained by formula (1):

[0081]

[0082] Where, x n Let x be the x-coordinate of the center of the middle circle of the finger-touched area corresponding to the distance set n. The first distance corresponding to the distance set n, For the second distance corresponding to the distance set n, D1 is the distance between the first sensor and the preset coordinate origin, and D2 is the preset distance between the first sensor and the second sensor.

[0083] In this embodiment, the upper left corner of the default projection area is the preset origin, with coordinates (0, 0). The coordinate axes are as follows: Figure 9 As shown.

[0084] The ordinate of the center of the finger-touching area corresponding to the distance set can be obtained by formula (2):

[0085]

[0086] Among them, y n Let l be the ordinate of the center of the finger-touched area corresponding to the distance set n, and l be the first distance or the second distance.

[0087] It should be noted that: in the embodiments of this application, l can be either the first distance or the second distance, and can be selected according to the actual situation. The embodiments of this application do not limit it here.

[0088] Step 1002: Based on the ordinate of the center of the finger touch area and the target distance, obtain the center radius of the finger touch area corresponding to the distance set, wherein the target distance is the distance between the first sensor and the second sensor; wherein the center radius of the finger touch area corresponding to the distance set can be obtained by formula (3):

[0089]

[0090] Where, r n Let n be the midpoint radius of the finger-touching area corresponding to the distance set n.

[0091] Step 1003: Obtain the radius of the finger touch area based on the midpoint radius of the finger touch area corresponding to each distance set.

[0092] In one embodiment, the average value of the median radius of the finger touch area corresponding to each distance set is determined as the radius of the finger touch area.

[0093] Step 805: Every third specified time interval, obtain the first distance and the second distance, and based on the radius of the finger touch area, the first distance, and the second distance, obtain the current position coordinates of the user's finger on the AR device display interface; wherein, the current horizontal position coordinates of the user's finger on the AR device display interface can be obtained by formula (4):

[0094]

[0095] Where Xm is the m-th current horizontal position coordinate of the user's finger on the AR device's display interface. For the m-th first distance, Let D1 be the distance between the first sensor and the preset coordinate origin, D2 be the preset distance between the first sensor and the second sensor, and N be a preset value.

[0096] It should be noted that N is 4 in this embodiment, but the value of N in this embodiment is not limited. The value of N is the same as the specified multiple mentioned above.

[0097] The current vertical coordinates of the user's finger on the AR device's display interface can be obtained using formula (5):

[0098]

[0099] Among them, Y m Let R be the m-th current vertical position coordinate of the user's finger on the AR device's display interface, and R be the radius of the area touched by the finger.

[0100] Step 806: Obtain the target sliding trajectory based on the current position coordinates.

[0101] In the embodiments of this application, the position points corresponding to each current position coordinate form the target sliding trajectory.

[0102] Step 306: If the interaction instruction is an interaction instruction of the user in the non-recording area, then control the display interface in the AR device based on the interaction instruction.

[0103] In this embodiment, the interaction commands for the non-recording area can be operations on the main display interface of the AR device. For example, when reading an article, a user can long-press the desktop and swipe up, down, left, or right to scroll through the article. When watching a video, a user can double-tap the non-recording area to pause or start the video.

[0104] To further understand the methods in this application, such as Figure 11 The diagram shown is a flowchart illustrating the interaction method of the AR device in this application, which may include the following steps:

[0105] Step 1101: Every second specified time interval, obtain the current pose of the mobile device;

[0106] Step 1102: Compare the current pose with the previous pose of the mobile device to determine if they are the same. If not, proceed to step 1103; if yes, proceed to step 1104.

[0107] Step 1103: Determine the target mode of the mobile device as the controller mode. Every first specified time interval, obtain the current pose of the mobile device. Based on the current pose of the mobile device, control the pose of the screen cursor in the AR device. When a target determination instruction sent by the user is received, use the target determination instruction to control the screen cursor in the AR device to complete the interactive operation.

[0108] Step 1104: Determine whether the number of times the current pose of the mobile device is obtained is greater than a specified number. If not, return to step 1101; if yes, proceed to step 1105.

[0109] Step 1105: Determine the target model of the mobile device as the touch mode;

[0110] Step 1106: Project the image using the miniature projector in the mobile device to obtain the projection area;

[0111] Step 1107: In response to the user's selection operation of each preset recording area in the projection area, the target recording area is obtained;

[0112] Step 1108: Determine the non-recording areas as the projection areas other than the target recording area;

[0113] Step 1109: If the interaction command is an input operation by the user in the recording area, when the user's finger begins to touch the recording area, the first sensor of the AR handle acquires a specified number of first distances;

[0114] Wherein, the first distance is the distance between the first sensor and the user's finger;

[0115] Step 1110: Obtain a specified number of second distances through the second sensor of the AR controller, wherein the second distance is the distance between the second sensor and the user's finger;

[0116] Step 1111: Based on the specified number of first distances and the specified number of second distances, obtain a specified number of distance sets, wherein any distance set includes the first distance and the second distance;

[0117] Step 1112: Based on a specified number of distance sets, obtain the radius of the finger-touched area;

[0118] Step 1113: Every third specified time interval, obtain the first distance and the second distance, and based on the radius of the finger touch area, the first distance and the second distance, obtain the current position coordinates of the user's finger on the display interface of the AR device;

[0119] Step 1114: Obtain the target sliding trajectory based on the current position coordinates, and overlay the target sliding trajectory on the display interface of the AR device;

[0120] Step 1115: If the interaction instruction is an interaction instruction of the user in the non-recording area, then control the display interface in the AR device based on the interaction instruction.

[0121] Based on the same inventive concept, the AR device interaction method described above can also be implemented by an AR device interaction device. The effect of this AR device interaction device is similar to that of the aforementioned method, and will not be described again here.

[0122] Figure 12 This is a schematic diagram of the structure of an interactive device for an AR device according to an embodiment of the present disclosure.

[0123] like Figure 12As shown, the interactive device 1200 of the AR device disclosed herein may include a target mode determination module 1210, a controller mode interaction module 1220, and a touch mode interaction module 1230.

[0124] The target mode determination module 1210 is used to determine the target mode based on the current pose of the mobile device in response to the interactive command sent by the user.

[0125] The controller mode interaction module 1220 is used to obtain the current pose of the mobile device every first specified time interval if the target mode is controller mode, control the pose of the screen cursor in the AR device based on the current pose of the mobile device, and control the screen cursor in the AR device to complete the interaction operation by using the target determination instruction sent by the user when the target determination instruction is received.

[0126] The touch mode interaction module 1230 is used to, if the target mode is touch mode, project through a micro projector in the mobile device to obtain a target recording area and a non-recording area; if the interaction command is an input operation by the user in the recording area, determine the target sliding trajectory in response to the input operation, and overlay the target sliding trajectory on the display interface in the AR device; if the interaction command is an interaction command by the user in the non-recording area, control the display interface in the AR device based on the interaction command.

[0127] In one embodiment, the target pattern determination module 1210 is specifically used for:

[0128] The current pose of the mobile device is obtained every second specified time interval;

[0129] Compare the current pose with the previous pose of the mobile device;

[0130] If the current pose is different from the previous pose, then the target mode of the mobile device is determined to be the controller mode; or,

[0131] If the current pose is the same as the previous pose, then determine whether the number of times the current pose of the mobile device is obtained is greater than a specified number.

[0132] If not, then return every second specified time interval to obtain the current pose of the mobile device, until it is determined that the current pose is different from the previous pose or the number of times the current pose of the mobile device has been obtained is greater than a specified number; or,

[0133] If so, then the target model of the mobile device is determined to be the touch mode.

[0134] In one embodiment, the touch mode interaction module 1230 performs the projection through the micro projector in the mobile device to obtain a target recording area and a non-recording area, specifically for:

[0135] The projection area is obtained by projecting using a miniature projector in the mobile device;

[0136] In response to the user's selection of preset recording areas in the projection area, the target recording area is obtained;

[0137] Other projection areas in the projection region besides the target recording area are defined as the non-recording areas.

[0138] In one embodiment, the touch mode interaction module 1230 performs the response to the input operation to determine the target sliding trajectory, specifically for:

[0139] When the user's finger begins to touch the recording area, the first sensor of the AR controller acquires a specified number of first distances, wherein the first distance is the distance between the first sensor and the user's finger; and

[0140] The AR controller's second sensor acquires a specified number of second distances, wherein the second distance is the distance between the second sensor and the user's finger;

[0141] Based on the specified number of first distances and the specified number of second distances, a specified number of distance sets are obtained, wherein any distance set includes the first distance and the second distance;

[0142] The radius of the finger-touched area is obtained based on a specified number of distance sets;

[0143] Every third specified time interval, the first distance and the second distance are obtained, and the current position coordinates of the user's finger on the display interface of the AR device are obtained based on the radius of the finger touch area, the first distance and the second distance;

[0144] The target sliding trajectory is obtained based on the current position coordinates.

[0145] In one embodiment, the touch mode interaction module 1230 executes the calculation based on a specified number of distance sets to obtain the radius of the finger touch area, specifically for:

[0146] For any set of distances, based on the first distance and the second distance in the set, the center of the finger touch area corresponding to the set is obtained; and,

[0147] Based on the ordinate of the center of the finger touch area and the target distance, the center radius of the finger touch area corresponding to the distance set is obtained, wherein the target distance is the distance between the first sensor and the second sensor;

[0148] The radius of the finger touch area is obtained based on the midpoint radius of the finger touch area corresponding to each distance set.

[0149] In one embodiment, the touch mode interaction module 1230 performs the step of obtaining the center of the finger touch area corresponding to the distance set based on the first distance and the second distance in the distance set, specifically for:

[0150] The x-coordinate of the center of the circle corresponding to the finger touch area of ​​the distance set can be obtained by the following formula:

[0151]

[0152] Where, x n Let x be the x-coordinate of the center of the middle circle of the finger-touched area corresponding to the distance set n. The first distance corresponding to the distance set n, For the second distance corresponding to the distance set n, D1 is the distance between the first sensor and the preset coordinate origin, and D2 is the preset distance between the first sensor and the second sensor;

[0153] The ordinate of the center of the circle corresponding to the finger touch area of ​​the distance set can be obtained by the following formula:

[0154]

[0155] Among them, y n Let l be the ordinate of the center of the middle circle of the finger-touched area corresponding to the distance set n, and l be the first distance or the second distance;

[0156] The touch mode interaction module 1230 executes the calculation based on the ordinate of the center of the finger touch area and the target distance to obtain the center radius of the finger touch area corresponding to the distance set, specifically for:

[0157] The midpoint radius of the finger touch area corresponding to the distance set can be obtained using the following formula:

[0158]

[0159] Where, r n Let n be the midpoint radius of the finger-touching area corresponding to the distance set n.

[0160] In one embodiment, the touch mode interaction module 1230 executes the step of obtaining the radius of the finger touch area based on the median radius of the finger touch area corresponding to each distance set, specifically for:

[0161] The average value of the median radius of the finger touch area corresponding to each distance set is determined as the radius of the finger touch area.

[0162] In one embodiment, the touch mode interaction module 1230 performs the step of obtaining the current position coordinates of the user's finger on the display interface of the AR device based on the radius of the finger touch area, the first distance, and the second distance, specifically for:

[0163] The current horizontal coordinates of the user's finger on the AR device's display interface are obtained using the following formula:

[0164]

[0165] Where Xm is the m-th current horizontal position coordinate of the user's finger on the AR device's display interface. For the m-th first distance, Let D1 be the distance between the first sensor and the preset coordinate origin, D2 be the preset distance between the first sensor and the second sensor, and N be a preset value.

[0166] The current vertical coordinates of the user's finger on the AR device's display interface are obtained using the following formula:

[0167]

[0168] Among them, Y m Let R be the m-th current vertical position coordinate of the user's finger on the AR device's display interface, and R be the radius of the area touched by the finger.

[0169] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as "circuit", "module", or "system".

[0170] In some possible implementations, various aspects of the AR device interaction method provided by the present invention can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in the AR device interaction method according to various exemplary embodiments of the present invention described above.

[0171] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An interaction method for an AR device, characterized in that, When applied to mobile devices, the method includes: In response to an interactive command sent by the user, a target mode is determined based on the current pose of the mobile device; If the target mode is the controller mode, the current pose of the mobile device is obtained every first specified time interval, the pose of the screen cursor in the AR device is controlled based on the current pose of the mobile device, and when a target determination instruction is received from the user, the screen cursor in the AR device is controlled to complete the interactive operation using the target determination instruction. If the target mode is a touch mode, a projector in the mobile device is used to project a target recording area and a non-recording area. If the interaction command is a user input operation in the recording area, when the user's finger begins to touch the recording area, a specified number of first distances are obtained through the first sensor of the mobile device, where the first distance is the distance between the first sensor and the user's finger; and a specified number of second distances are obtained through the second sensor of the mobile device, where the second distance is the distance between the second sensor and the user's finger. A specified number of distance sets are obtained based on the specified number of first distances and the specified number of second distances, where any distance set includes the first distance and the second distance. The radius of the finger-touched area is obtained based on the specified number of distance sets. Every third specified time interval, the first distance and the second distance are obtained, and the current position coordinates of the user's finger on the display interface of the AR device are obtained based on the radius of the finger-touched area, the first distance, and the second distance. A target sliding trajectory is obtained based on each current position coordinate, and the target sliding trajectory is superimposed and displayed on the display interface of the AR device. If the interaction command is an interaction command from the user in the non-recording area, the display interface of the AR device is controlled based on the interaction command.

2. The method according to claim 1, characterized in that, Determining the target pattern based on the current pose of the mobile device includes: The current pose of the mobile device is obtained every second specified time interval; Compare the current pose with the previous pose of the mobile device; If the current pose is different from the previous pose, then the target mode of the mobile device is determined to be the controller mode; or, If the current pose is the same as the previous pose, then determine whether the number of times the current pose of the mobile device is obtained is greater than a specified number. If not, then return every second specified time interval to obtain the current pose of the mobile device, until it is determined that the current pose is different from the previous pose or the number of times the current pose of the mobile device has been obtained is greater than a specified number; or, If so, then the target model of the mobile device is determined to be the touch mode.

3. The method according to claim 1, characterized in that, The process of projecting through a micro projector in the mobile device to obtain a target recording area and a non-recording area includes: The projection area is obtained by projecting using a miniature projector in the mobile device; In response to the user's selection of preset recording areas in the projection area, the target recording area is obtained; Other projection areas in the projection region besides the target recording area are defined as the non-recording areas.

4. The method according to claim 3, characterized in that, The radius of the finger-touched area is obtained based on a specified number of distance sets, including: For any set of distances, based on the first distance and the second distance in the set, the center of the finger touch area corresponding to the set is obtained; and, Based on the ordinate of the center of the finger touch area and the target distance, the center radius of the finger touch area corresponding to the distance set is obtained, wherein the target distance is the distance between the first sensor and the second sensor; The radius of the finger touch area is obtained based on the midpoint radius of the finger touch area corresponding to each distance set.

5. The method according to claim 4, characterized in that, The step of obtaining the center of the finger touch area corresponding to the distance set based on the first distance and the second distance in the distance set includes: The x-coordinate of the center of the circle corresponding to the finger touch area of ​​the distance set can be obtained by the following formula: ； in, Let x be the x-coordinate of the center of the middle circle of the finger-touched area corresponding to the distance set n. The first distance corresponding to the distance set n, The second distance corresponding to the distance set n, The distance between the first sensor and the preset coordinate origin is denoted as . The preset distance between the first sensor and the second sensor; The ordinate of the center of the circle corresponding to the finger touch area of ​​the distance set can be obtained by the following formula: ； in, Let be the ordinate of the center of the circle in the finger-touched area corresponding to the distance set n. It can be either the first distance or the second distance; The process of obtaining the center radius of the finger touch area corresponding to the set of distances, based on the ordinate of the center circle of the finger touch area and the target distance, includes: The midpoint radius of the finger touch area corresponding to the distance set can be obtained using the following formula: ； in, Let n be the midpoint radius of the finger-touching area corresponding to the distance set n.

6. The method according to claim 4, characterized in that, The step of obtaining the radius of the finger touch area based on the median radius of the finger touch area corresponding to each distance set includes: The average value of the median radius of the finger touch area corresponding to each distance set is determined as the radius of the finger touch area.

7. The method according to claim 1, characterized in that, The step of obtaining the current position coordinates of the user's finger on the display interface of the AR device based on the radius of the finger touch area, the first distance, and the second distance includes: The current horizontal coordinates of the user's finger on the AR device's display interface are obtained using the following formula: ； in, Let m be the coordinates of the current horizontal position of the user's finger on the display interface of the AR device. For the m-th first distance, For the m-th second distance, The distance between the first sensor and the preset coordinate origin is denoted as . N is a preset distance between the first sensor and the second sensor; The current vertical coordinates of the user's finger on the AR device's display interface are obtained using the following formula: ； in, Let m be the current vertical coordinate of the user's finger on the display interface of the AR device. The radius of the area touched by the finger.

8. A mobile device, characterized in that, It includes a processor and a memory, which are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: In response to an interactive command sent by the user, a target mode is determined based on the current pose of the mobile device; If the target mode is the controller mode, then every first specified time interval, the current pose of the mobile device is obtained, the pose of the screen cursor in the AR device is controlled based on the current pose of the mobile device, and when a target determination instruction is received from the user, the target determination instruction is used to control the screen cursor in the AR device to complete the interactive operation. If the target mode is a touch mode, a projector in the mobile device is used to project a target recording area and a non-recording area. If the interaction command is a user input operation in the recording area, when the user's finger begins to touch the recording area, a specified number of first distances are obtained through the first sensor of the mobile device, where the first distance is the distance between the first sensor and the user's finger; and a specified number of second distances are obtained through the second sensor of the mobile device, where the second distance is the distance between the second sensor and the user's finger. A specified number of distance sets are obtained based on the specified number of first distances and the specified number of second distances, where any distance set includes the first distance and the second distance. The radius of the finger-touched area is obtained based on the specified number of distance sets. Every third specified time interval, the first distance and the second distance are obtained, and the current position coordinates of the user's finger on the display interface of the AR device are obtained based on the radius of the finger-touched area, the first distance, and the second distance. A target sliding trajectory is obtained based on each current position coordinate, and the target sliding trajectory is superimposed and displayed on the display interface of the AR device. If the interaction command is an interaction command from the user in the non-recording area, the display interface of the AR device is controlled based on the interaction command.

9. The mobile device according to claim 8, characterized in that, The processor is specifically configured to perform the projection through the micro-projector in the mobile device to obtain a target recording area and a non-recording area, specifically as follows: The projection area is obtained by projecting using a miniature projector in the mobile device; In response to the user's selection of preset recording areas in the projection area, the target recording area is obtained; Other projection areas in the projection region besides the target recording area are defined as the non-recording areas.

Images