Interaction method of AR (Augmented Reality) equipment and mobile equipment

Abstract

The invention provides an interaction method of AR equipment and mobile equipment, which are used for providing multiple interaction modes and improving user experience. Comprising the steps of determining a target mode based on a current pose of mobile equipment; if the target mode is a handle mode, the current pose of the mobile device is obtained every first specified duration, the pose of a screen cursor in the AR device is controlled based on the current pose, and when a target determination instruction is received, the screen cursor of the AR device is controlled by using the target determination instruction to complete interaction operation; if the target mode is a touch mode, performing projection through a miniature projector in the mobile equipment to obtain a target recording area and a non-recording area; if the interaction instruction is an input operation of the user in the recording area, determining a target sliding track in response to the input operation, and displaying the target sliding track in a display interface in the AR equipment in an overlapping manner; and if the interaction instruction is the interaction instruction of the user in the non-recording area, controlling a display interface in the AR equipment based on the interaction instruction.

Description

Background Art

[0002] As an emerging wearable device, the design of the operation and interaction mode of AR (Augmented Reality) glasses is crucial to the user experience. Currently, the most commonly used interaction mode is to interact with AR glasses through handheld controllers. Handheld controllers used with AR glasses are similar to game controllers and can provide more precise and complex operations.

[0003] However, the handheld controller in the prior art only interacts with the AR glasses by moving the corresponding controller held by the user, and the functions are relatively simple, resulting in a poor user interaction experience. Summary of the invention

[0004] The present application provides an AR device interaction method and a mobile device, which are used to provide multiple interaction modes with the AR device to improve user experience.

[0005] In a first aspect, an embodiment of the present application provides an interaction method of an object AR device, the method comprising:

[0006] In response to an interaction instruction sent by a user, determining a target mode based on a current position of the mobile device;

[0007] If the target mode is the handle mode, then every first specified time period, the current position of the mobile device is obtained, the position of the screen cursor in the AR device is controlled based on the current position of the mobile device, and when a target determination instruction sent by the user is received, the screen cursor in the AR device is controlled by using the target determination instruction to complete the interactive operation;

[0008] If the target mode is the touch mode, projection is performed through the micro-projector in the mobile device to obtain a target recording area and a non-recording area; if the interaction instruction is an input operation of the user in the recording area, the target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed in the display interface of the AR device; if the interaction instruction is an interaction instruction of the user in the non-recording area, the display interface in the AR device is controlled based on the interaction instruction.

[0009] A second aspect of the present application provides a mobile device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0010] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0011] In response to an interaction instruction sent by a user, determining a target mode based on a current position of the mobile device;

[0012] If the target mode is the handle mode, then every first specified time period, the current position of the mobile device is obtained, the position of the screen cursor in the AR device is controlled based on the current position of the mobile device, and when a target determination instruction sent by the user is received, the screen cursor in the AR device is controlled by using the target determination instruction to complete the interactive operation;

[0013] If the target mode is the touch mode, projection is performed through the micro-projector in the mobile device to obtain a target recording area and a non-recording area; if the interaction instruction is an input operation of the user in the recording area, the target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed in the display interface of the AR device; if the interaction instruction is an interaction instruction of the user in the non-recording area, the display interface in the AR device is controlled based on the interaction instruction.

[0014] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.

[0015] In the above-mentioned embodiments of the present application, the target mode is determined based on the current posture of the mobile device in response to the interaction instruction sent by the user; if the target mode is the handle mode, the current posture of the mobile device is obtained every first specified time, and the posture of the screen cursor in the AR device is controlled based on the current posture of the mobile device, and when the target determination instruction sent by the user is received, the screen cursor in the AR device is controlled by the target determination instruction to complete the interaction operation; if the target mode is the touch mode, the micro projector in the mobile device is used for projection to obtain the target recording area and the non-recording area; if the interaction instruction is the input operation of the user in the recording area, the target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed in the display interface of the AR device; if the interaction instruction is the interaction instruction of the user in the non-recording area, the display interface in the AR device is controlled based on the interaction instruction. Therefore, a variety of interaction modes are provided in the embodiments of the present application, which expands the AR interaction mode and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0018] Figure 2 The structural diagram of the mobile device provided by the embodiment of the present application is exemplarily shown;

[0019] Figure 3 One of the flowcharts of the interaction method of the AR device provided in the embodiment of the present application is exemplarily shown;

[0020] Figure 4 The following is a schematic diagram of a process for determining a target mode provided by an embodiment of the present application;

[0021] Figure 5 The schematic diagram of the handle mode provided by the embodiment of the present application is exemplarily shown;

[0022] Figure 6 A schematic diagram exemplarily shows a touch mode provided in an embodiment of the present application;

[0023] Fig. 7A A schematic diagram exemplarily shows a recording area provided in an embodiment of the present application;

[0024] Figure 7B The overlay display schematic diagram provided in the embodiment of the present application is exemplarily shown;

[0025] Figure 8 The flowchart for determining the target sliding trajectory provided by the embodiment of the present application is exemplarily shown;

[0026] Fig. 9 The calibration schematic diagram in the projection area provided by the embodiment of the present application is exemplarily shown;

[0027] Fig.10 The schematic diagram of the process of determining the radius of the finger touch area provided in the embodiment of the present application is exemplarily shown;

[0028] Fig.11 The second flowchart of the interaction method of the AR device provided in the embodiment of the present application is exemplarily shown;

[0029] Fig.12 The structure diagram of the interaction device of the AR device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0030] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0031] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0032] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0033] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0034] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0035] The idea of ​​the embodiments of the present application is summarized below.

[0036] The current handheld controllers only interact with AR glasses by moving the corresponding controllers held by the user. The functions are relatively simple, resulting in a poor user interaction experience.

[0037] Based on the problem of poor user interaction experience caused by the single interactive function in the prior art. The embodiment of the present application provides an interactive method for an AR device, which determines the target mode based on the current posture of the mobile device in response to the interactive instruction sent by the user; if the target mode is the handle mode, the current posture of the mobile device is obtained every first specified time, and the posture of the screen cursor in the AR device is controlled based on the current posture of the mobile device, and when the target determination instruction sent by the user is received, the screen cursor in the AR device is controlled by the target determination instruction to complete the interactive operation; if the target mode is the touch mode, the micro projector in the mobile device is used for projection to obtain the target recording area and the non-recording area; if the interactive instruction is the input operation of the user in the recording area, the target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed in the display interface of the AR device; if the interactive instruction is the interactive instruction of the user in the non-recording area, the display interface in the AR device is controlled based on the interactive instruction. Therefore, a variety of interactive modes are provided in the embodiment of the present application, which expands the AR interaction mode and improves the user experience.

[0038] like Figure 1 As shown, an application scenario of an interaction method of an AR device includes a mobile device 101 and an AR device 102.

[0039] In a possible application scenario, the mobile device 101 determines a target mode based on a current position of the mobile device 101 in response to an interaction instruction sent by a user; if the target mode is a handle mode, the mobile device 101 obtains the current position of the mobile device every first specified time period, controls the position of a screen cursor in the AR device 102 based on the current position of the mobile device, and when receiving a target determination instruction sent by the user, controls the screen cursor in the AR device 102 using the target determination instruction to complete the interaction operation; if the target mode is a touch mode, the mobile device 101 projects through a micro-projector in the mobile device 101 to obtain a target recording area and a non-recording area; if the interaction instruction is an input operation of the user in the recording area, a target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed in a display interface in the AR device 102; if the interaction instruction is an interaction instruction of the user in the non-recording area, the display interface in the AR device 102 is controlled based on the interaction instruction.

[0040] in, Figure 1 The mobile device 101 and the AR device 102 can exchange information through a communication network, wherein the communication mode adopted by the communication network can be divided into a wireless communication mode or a wired communication mode.

[0041] Exemplarily, the mobile device 101 can access the network through cellular mobile communication technology to communicate with the AR device 102, wherein the cellular mobile communication technology, for example, includes the fifth generation mobile communication (5th Generation Mobile Networks, 5G) technology.

[0042] Optionally, the mobile device 101 may access the network and communicate with the AR device 102 via a short-range wireless communication method, wherein the short-range wireless communication method, for example, includes Wireless Fidelity (Wi-Fi) technology.

[0043] Among them, the description in this application only details a single mobile device 101 and a single AR device 102, but those skilled in the art should understand that the mobile device 101 and the single AR device 102 shown are intended to represent the operations of the mobile device 101 and the AR device 102 involved in the technical solution of this application. It does not imply any restrictions on the number, type or location of the mobile devices 101 and the AR devices 102. It should be noted that if additional modules are added to the illustrated environment or individual modules are removed therefrom, the underlying concepts of the example embodiments of this application will not be changed.

[0044] It should be noted that the AR device interaction method proposed in this application is not only applicable to Figure 1 The application scenario shown is also applicable to any interactive device with an AR device.

[0045] The following describes the interaction method of the AR device in the exemplary implementation of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the methods and principles of the present application, and the implementation of the present application is not limited in this respect.

[0046] Before introducing the interactive method of the AR device in the present application, the mobile device in the embodiment of the present application is first introduced. Figure 2 As shown, it is a schematic diagram of the structure of the mobile device in the embodiment of the present application, from Figure 2 It can be seen that the mobile device 200 in the embodiment of the present application includes a micro projector 201, a touch panel 202, a first sensor 203, a second sensor 204, a system prompt light 205, a processor 206, a holder 207, a memory 208, an IMU sensor 209 and a Bluetooth module 210. Among them:

[0047] The memory 208 stores a computer program, and the processor 206 is configured to perform the following operations based on the computer program: in response to an interaction instruction sent by the user through the touch panel 202, determine a target mode based on the current position of the mobile device obtained by the IMU sensor 209; if the target mode is the handle mode, obtain the current position of the mobile device at a first specified time interval, and send the current position of the mobile device to the AR device through the Bluetooth module 210, so as to control the position of the screen cursor in the AR device based on the current position of the mobile device, and when receiving a target determination instruction sent by the user, use the target determination instruction to control the screen cursor in the AR device to complete the interaction operation; if the target mode is the handle mode, obtain the current position of the mobile device at a first specified time interval, and send the current position of the mobile device to the AR device through the Bluetooth module 210, so as to control the position of the screen cursor in the AR device based on the current position of the mobile device, and when receiving a target determination instruction sent by the user, use the target determination instruction to control the screen cursor in the AR device to complete the interaction operation; If the target mode is the touch mode, projection is performed through the micro-projector 201 in the mobile device to obtain a target recording area and a non-recording area; if the interaction instruction is an input operation of the user in the recording area, in response to the input operation, a first distance between the first sensor and the user's finger is obtained through the first sensor 203, and a second distance between the second sensor and the user's finger is obtained through the second sensor 204, and a target sliding trajectory is determined based on the first distance and the second distance, and the target sliding trajectory is superimposed and displayed in the display interface of the AR device; if the interaction instruction is an interaction instruction of the user in the non-recording area, the display interface in the AR device is controlled based on the interaction instruction.

[0048] The system prompt light 205 is used to send a prompt signal to the user, and the fixer 207 is used to fix the mobile device.

[0049] The following is an introduction to the interaction method of the AR device in the embodiment of the present application with reference to the accompanying drawings. Figure 3 FIG. 1 is a flow chart of an interaction method of an AR device, which may specifically include the following steps:

[0050] Step 301: In response to an interaction instruction sent by a user, determining a target mode based on a current position of the mobile device;

[0051] Next, the method of determining the target mode in the embodiment of the present application is described. Figure 4 As shown, it is a schematic diagram of a process for determining a target mode, which may specifically include the following steps:

[0052] Step 401: Obtaining the current position and posture of the mobile device every second specified time period;

[0053] The second specified time length in the embodiment of the present application can be set according to actual conditions, but the embodiment of the present application does not limit the specific value of the second specified time length.

[0054] Step 402: Compare the current posture with the previous posture of the mobile device to determine whether they are the same, if not, execute step 403, if they are the same, execute step 404;

[0055] Step 403: Determine that the target mode of the mobile device is the handle mode;

[0056] Step 404: determine whether the number of times the current position of the mobile device is obtained is greater than a specified number, if not, return to step 401, if yes, execute step 405;

[0057] Step 405: Determine that the target model of the mobile device is the touch mode.

[0058] It should be noted that the specified number in the embodiment of the present application is 10, but the specific value of the specified number in the embodiment of the present application is not limited. The specific value of the specified number in the embodiment of the present application can be set according to actual conditions.

[0059] Step 302: if the target mode is the handle mode, obtaining the current position of the mobile device every first specified time period, and controlling the position of the screen cursor in the AR device based on the current position of the mobile device;

[0060] like Figure 5 As shown, in the handle mode in the embodiment of the present application, the mobile device in the embodiment of the present application can be worn on the wrist of any arm of the user. When the user swings his arms up and down, left and right, the mobile device obtains its current posture and transmits the current posture to the AR device through the Bluetooth module. The AR device receives the current posture of the mobile device that is continuously updated with the movement of the arm, determines the posture of the screen cursor in the AR device based on the current posture of the mobile device, and uses the determined posture of the screen cursor to control the posture of the screen cursor in the AR device.

[0061] In one embodiment, controlling the position of the screen cursor in the AR device based on the current position of the mobile device may be specifically implemented as follows:

[0062] By utilizing the pre-set correspondence between the posture of the mobile device and the posture of the screen cursor, based on the current posture of the mobile device, the target posture of the screen cursor corresponding to the current posture of the mobile device is obtained, and the posture of the screen cursor in the AR device is controlled to move to the target posture.

[0063] Step 303: When receiving a target determination instruction sent by the user, using the target determination instruction to control a screen cursor in the AR device to complete an interactive operation;

[0064] The purpose of swinging the arm to control the change in the direction of the screen cursor in the AR device in the embodiment of the present application is to adjust the direction of the screen cursor. When the screen cursor points to the desired target, it is necessary to confirm the selection of the pointed target by clicking, and at this time, the confirmation operation can be performed by the other hand that is not wearing the mobile device. The specific operation method is: use any finger of the hand to touch the touch panel of the mobile device. When the touch time exceeds 0.1s, the system of the mobile device can recognize that the current touch is valid, and the system prompt light of the mobile device flashes blue. The mobile device then transmits the touch signal to the AR device through the Bluetooth module. After receiving the touch signal, the AR device recognizes it as a confirmation signal, thereby completing the confirmation operation of the screen cursor pointing to the target to complete the interactive operation.

[0065] Step 304: if the target mode is the touch mode, projecting is performed through the micro-projector in the mobile device to obtain a target recording area and a non-recording area;

[0066] like Figure 6 As shown, it is a schematic diagram of the touch mode. The mobile device in the embodiment of the present application can be placed upright on any horizontal desktop in touch mode. And a projection area can be projected on the horizontal desktop. The projection area in the embodiment of the present application is consistent with the resolution of the AR device, and the area size in the embodiment of the present application is enlarged according to the screen size of the AR device according to the specified multiple. The specified multiple in the embodiment of the present application is 4 times, but the specified multiple is not limited. In the projection area, the user can write text by sliding his finger on the desktop. The final written text information is rendered in the upper screen of the video, game or application screen through the overlay of the AR glasses to record the information needed by the user during use.

[0067] In one embodiment, step 304 can be specifically implemented as follows: projecting through the micro-projector in the mobile device to obtain a projection area; in response to the user's selection operation on each recording area preset in the projection area, obtaining the target recording area; and determining other projection areas in the projection area except the target recording area as the non-recording area.

[0068] The micro projector in the embodiment of the present application projects a specified number of preset recording areas in the projection area. Fig. 7A As shown, Fig. 7A The white area in the image is the preset recording area. The user can select one or more recording areas by sliding or clicking to determine the target recording area. At this time, the target recording area selected by the user turns gray. At this time, the target recording area selected by the user will be displayed on the surface of the video, game or application through overlay on the screen of the AR device. The specific overlay display method is as follows: Figure 7B As shown. At this time, if the user finds that the target recording area affects the use and wants to change the recording area, he only needs to click the selected target recording area again, and the recording area will be canceled and turn yellow again. If the user does not touch the projection area for more than 1 minute, the default target recording area selection phase ends, and the target recording area in the AR device is determined. After the target recording area is determined, the target recording area selected by the user in the projection area turns white (target recording area), and the unselected recording area turns black (non-recording area).

[0069] Step 305: if the interaction instruction is an input operation of the user in the recording area, in response to the input operation, determining a target sliding trajectory, and superimposing and displaying the target sliding trajectory in a display interface of the AR device;

[0070] Next, the method of determining the target sliding trajectory in step 305 is introduced. Figure 8 As shown, it is a schematic diagram of a process for determining a target sliding trajectory, which may specifically include the following steps:

[0071] Step 801: When the user's finger starts to touch the recording area, a first distance of a specified number is acquired through a first sensor of the AR handle, wherein the first distance is the distance between the first sensor and the user's finger;

[0072] In the embodiment of the present application, the user's finger initially touching the recording area may be the user's first touching of the recording area, or the user may touch the recording area, then leave the recording area, and then touch the recording area again.

[0073] If the user touches the recording area for the first time, the user can place the touching finger on the calibration point in the projection area, as shown in the figure. Fig. 9 As shown, each placement time of the user is about 1 second, the number of placements is 10 times, the first sensor and the second sensor collect the first distance and the second distance respectively, after each successful collection, the system indicator light turns green, and the red light turns red when the collection fails, and the system prompt light goes out after the collection is completed.

[0074] Step 802: Acquire a specified number of second distances through a second sensor of the AR handle, wherein the second distance is a distance between the second sensor and the user's finger;

[0075] It should be noted that the embodiment of the present application does not limit the execution sequence of step 801 and step 802. Step 801 may be executed first, and then step 802; step 802 may be executed first, and then step 801; or step 801 and step 802 may be executed simultaneously.

[0076] Step 803: obtaining a specified number of distance sets according to the specified number of first distances and the specified number of second distances, wherein any distance set includes the first distance and the second distance;

[0077] The first distance and the second distance in any distance set in the embodiments of the present application are acquired at the same time point.

[0078] Step 804: based on a specified number of distance sets, obtain the radius of the finger touch area;

[0079] like Fig.10 As shown, it is a schematic diagram of a process of determining the radius of the finger touch area, which may specifically include the following steps:

[0080] Step 1001: for any distance set, according to the first distance and the second distance in the distance set, obtain the middle center of the finger touch area corresponding to the distance set; wherein the abscissa of the middle center of the finger touch area corresponding to the distance set can be obtained by formula (1):

[0081]

[0082] Among them, x n is the horizontal coordinate of the center of the middle circle of the finger touch area corresponding to the distance set n, is the first distance corresponding to the distance set n, is a second distance corresponding to the distance set n, D1 is a distance from the first sensor to a preset coordinate origin, and D2 is a preset distance between the first sensor and the second sensor.

[0083] In the embodiment of the present application, the upper left corner of the projection area is assumed to be the preset coordinate origin, and the coordinate value is (0, 0). Fig. 9 shown.

[0084] The ordinate of the center of the middle circle of the finger touch area corresponding to the distance set can be obtained by formula (2):

[0085]

[0086] Among them, y n is the ordinate of the middle circle center of the finger touch area corresponding to the distance set n, and l is the first distance or the second distance.

[0087] It should be noted that: l in the embodiment of the present application can be either the first distance or the second distance, which can be selected according to actual conditions, and the embodiment of the present application is not limited here.

[0088] Step 1002: Based on the ordinate of the middle circle center of the finger touch area and the target distance, obtain the middle radius of the finger touch area corresponding to the distance set, wherein the target distance is the distance between the first sensor and the second sensor; wherein the middle radius of the finger touch area corresponding to the distance set can be obtained by formula (3):

[0089]

[0090] Among them, r n is the middle radius of the finger touch area corresponding to the distance set n.

[0091] Step 1003: Obtain the radius of the finger touch area according to the middle radius of the finger touch area corresponding to each distance set.

[0092] In one embodiment, the average value of the middle radii of the finger touch areas corresponding to the distance sets is determined as the radius of the finger touch area.

[0093] Step 805: every third specified time period, the first distance and the second distance are obtained, and the current position coordinates of the user's finger on the display interface of the AR device are obtained according to the radius of the finger touch area, the first distance and the second distance; wherein the current horizontal position coordinates of the user's finger on the display interface of the AR device can be obtained by formula (4):

[0094]

[0095] Wherein, Xm is the mth current horizontal position coordinate of the user's finger in the display interface of the AR device, is the mth first distance, is the mth second distance, D1 is the distance between the first sensor and a preset coordinate origin, D2 is the preset distance between the first sensor and the second sensor, and N is a preset value.

[0096] It should be noted that: N in the embodiment of the present application is 4, but the value of N in the embodiment of the present application is not limited. The value of N is the same as the value of the specified multiple described above.

[0097] The current vertical position coordinate of the user's finger on the display interface of the AR device can be obtained by formula (5):

[0098]

[0099] Among them, Y m is the mth current vertical position coordinate of the user's finger on the display interface of the AR device, and R is the radius of the finger touch area.

[0100] Step 806: Obtain the target sliding trajectory according to the current position coordinates.

[0101] In the embodiment of the present application, the position points corresponding to the current position coordinates constitute the target sliding trajectory.

[0102] Step 306: If the interaction instruction is an interaction instruction of the user in the non-recording area, controlling a display interface in the AR device based on the interaction instruction.

[0103] In the embodiment of the present application, the interactive instruction to the non-recording area can be an operation on the main display interface of the AR device. For example, when a user is watching an article, he can slide the article up, down, left, and right by long pressing the desktop. When a user is watching a video, he can pause and start the video by double-clicking the non-recording area.

[0104] In order to further understand the method in this application, Fig.11 FIG. 1 is a flow chart of the interaction method of the AR device in the present application, which may include the following steps:

[0105] Step 1101: Obtain the current position and posture of the mobile device every second specified time period;

[0106] Step 1102: Compare the current posture with the previous posture of the mobile device to determine whether they are the same, if not, execute step 1103, if yes, execute step 1104;

[0107] Step 1103: determining that the target mode of the mobile device is the handle mode, obtaining the current position of the mobile device at intervals of a first specified time, controlling the position of the screen cursor in the AR device based on the current position of the mobile device, and when receiving a target determination instruction sent by the user, controlling the screen cursor in the AR device using the target determination instruction to complete the interactive operation;

[0108] Step 1104: determine whether the number of times the current position of the mobile device is obtained is greater than a specified number, if not, return to step 1101, if yes, execute step 1105;

[0109] Step 1105: Determine that the target model of the mobile device is the touch mode;

[0110] Step 1106: Projecting by the micro-projector in the mobile device to obtain a projection area;

[0111] Step 1107: in response to the user's selection operation on each preset recording area in the projection area, obtaining the target recording area;

[0112] Step 1108: determining other projection areas in the projection area except the target recording area as the non-recording area;

[0113] Step 1109: if the interaction instruction is an input operation of the user in the recording area, when the user's finger begins to touch the recording area, a first distance of a specified number is acquired through the first sensor of the AR handle;

[0114] Wherein, the first distance is the distance between the first sensor and the user's finger;

[0115] Step 1110: Acquire a specified number of second distances through a second sensor of the AR handle, wherein the second distance is a distance between the second sensor and the user's finger;

[0116] Step 1111: obtaining a specified number of distance sets according to the specified number of first distances and the specified number of second distances, wherein any distance set includes the first distance and the second distance;

[0117] Step 1112: based on a specified number of distance sets, obtain the radius of the finger touch area;

[0118] Step 1113: acquiring the first distance and the second distance every third specified time period, and obtaining the current position coordinates of the user's finger on the display interface of the AR device according to the radius of the finger touch area, the first distance, and the second distance;

[0119] Step 1114: obtaining the target sliding trajectory according to each current position coordinate, and superimposing and displaying the target sliding trajectory in the display interface of the AR device;

[0120] Step 1115: If the interaction instruction is an interaction instruction of the user in the non-recording area, controlling a display interface in the AR device based on the interaction instruction.

[0121] Based on the same inventive concept, the AR device interaction method disclosed above can also be implemented by an AR device interaction device. The effect of the AR device interaction device is similar to that of the aforementioned method, and will not be repeated here.

[0122] Fig.12 The figure is a schematic diagram of the structure of an interaction device of an AR device according to an embodiment of the present disclosure.

[0123] like Fig.12As shown, the interaction device 1200 of the AR device of the present disclosure may include a target mode determination module 1210 , a handle mode interaction module 1220 , and a touch mode interaction module 1230 .

[0124] A target mode determination module 1210 is used to determine a target mode based on a current position of the mobile device in response to an interaction instruction sent by a user;

[0125] The handle mode interaction module 1220 is configured to obtain the current position of the mobile device at intervals of a first specified time if the target mode is the handle mode, control the position of the screen cursor in the AR device based on the current position of the mobile device, and control the screen cursor in the AR device using the target determination instruction to complete the interaction operation when receiving a target determination instruction sent by the user;

[0126] The touch mode interaction module 1230 is used to, if the target mode is the touch mode, perform projection through the micro-projector in the mobile device to obtain a target recording area and a non-recording area; if the interaction instruction is an input operation of the user in the recording area, then in response to the input operation, determine the target sliding trajectory, and superimpose and display the target sliding trajectory in the display interface of the AR device; if the interaction instruction is an interaction instruction of the user in the non-recording area, then control the display interface in the AR device based on the interaction instruction.

[0127] In one embodiment, the target mode determination module 1210 is specifically configured to:

[0128] Every second specified time period, obtaining the current position and posture of the mobile device;

[0129] comparing the current pose with a previous pose of the mobile device;

[0130] If the current posture is different from the previous posture, determining that the target mode of the mobile device is the handle mode; or,

[0131] If the current posture is the same as the previous posture, determining whether the number of times the current posture of the mobile device is obtained is greater than a specified number;

[0132] If not, return to obtaining the current position and posture of the mobile device every second specified time period until it is determined that the current position and posture are different from the previous position and posture or the number of times the current position and posture of the mobile device are obtained is greater than a specified number; or,

[0133] If yes, it is determined that the target model of the mobile device is the touch mode.

[0134] In one embodiment, the touch mode interaction module 1230 performs the projection through the micro projector in the mobile device to obtain the target recording area and non-recording area, which is specifically used for:

[0135] Projecting by means of a micro-projector in the mobile device to obtain a projection area;

[0136] In response to a user's selection operation on each recording area preset in the projection area, obtaining the target recording area;

[0137] The other projection areas in the projection area except the target recording area are determined as the non-recording areas.

[0138] In one embodiment, the touch mode interaction module 1230 performs the step of determining the target sliding track in response to the input operation, specifically for:

[0139] When the user's finger begins to touch the recording area, a first distance of a specified number is acquired through a first sensor of the AR handle, wherein the first distance is a distance between the first sensor and the user's finger; and

[0140] Acquire, by means of a second sensor of the AR handle, a second distance of a specified number, wherein the second distance is a distance between the second sensor and a finger of the user;

[0141] Obtaining a specified number of distance sets according to the specified number of first distances and the specified number of second distances, wherein any one of the distance sets includes the first distance and the second distance;

[0142] Based on the specified number of distance sets, get the radius of the finger touch area;

[0143] At every third specified time period, the first distance and the second distance are acquired, and according to the radius of the finger touch area, the first distance and the second distance, the current position coordinates of the user's finger on the display interface of the AR device are obtained;

[0144] According to each current position coordinate, the target sliding trajectory is obtained.

[0145] In one embodiment, the touch mode interaction module 1230 executes the distance set based on the specified number to obtain the radius of the finger touch area, specifically for:

[0146] For any distance set, according to the first distance and the second distance in the distance set, the middle center of the finger touch area corresponding to the distance set is obtained; and

[0147] Based on the ordinate of the middle circle center of the finger touch area and the target distance, obtaining the middle radius of the finger touch area corresponding to the distance set, wherein the target distance is the distance between the first sensor and the second sensor;

[0148] The radius of the finger touch area is obtained according to the middle radius of the finger touch area corresponding to each distance set.

[0149] In one embodiment, the touch mode interaction module 1230 performs the step of obtaining the middle center of the finger touch area corresponding to the distance set according to the first distance and the second distance in the distance set, specifically for:

[0150] The abscissa of the center of the middle circle of the finger touch area corresponding to the distance set is obtained by the following formula:

[0151]

[0152] Among them, x n is the horizontal coordinate of the center of the middle circle of the finger touch area corresponding to the distance set n, is the first distance corresponding to the distance set n, is a second distance corresponding to the distance set n, D1 is a distance between the first sensor and a preset coordinate origin, and D2 is a preset distance between the first sensor and the second sensor;

[0153] The ordinate of the center of the middle circle of the finger touch area corresponding to the distance set is obtained by the following formula:

[0154]

[0155] Among them, y n is the ordinate of the middle circle center of the finger touch area corresponding to the distance set n, and l is the first distance or the second distance;

[0156] The touch mode interaction module 1230 executes the ordinate based on the middle circle center of the finger touch area and the target distance to obtain the middle radius of the finger touch area corresponding to the distance set, specifically for:

[0157] The middle radius of the finger touch area corresponding to the distance set is obtained by the following formula:

[0158]

[0159] Among them, r n is the middle radius of the finger touch area corresponding to the distance set n.

[0160] In one embodiment, the touch mode interaction module 1230 executes the middle radius of the finger touch area corresponding to each distance set to obtain the radius of the finger touch area, which is specifically used for:

[0161] The average value of the middle radii of the finger touch areas corresponding to the distance sets is determined as the radius of the finger touch area.

[0162] In one embodiment, the touch mode interaction module 1230 performs the step of obtaining the current position coordinates of the user's finger in the display interface of the AR device according to the radius of the finger touch area, the first distance, and the second distance, specifically for:

[0163] The current horizontal position coordinates of the user's finger on the display interface of the AR device are obtained by the following formula:

[0164]

[0165] Wherein, Xm is the mth current horizontal position coordinate of the user's finger in the display interface of the AR device, is the mth first distance, is the mth second distance, D1 is the distance between the first sensor and a preset coordinate origin, D2 is the preset distance between the first sensor and the second sensor, and N is a preset value;

[0166] The current vertical position coordinate of the user's finger on the display interface of the AR device is obtained by the following formula:

[0167]

[0168] Among them, Y m is the mth current vertical position coordinate of the user's finger on the display interface of the AR device, and R is the radius of the finger touch area.

[0169] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0170] In some possible implementations, various aspects of the interaction method of an AR device provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the interaction method of an AR device according to various exemplary embodiments of the present invention described above in this specification.

[0171] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An interactive method for an AR device, characterized in that: Applied in a mobile device, the method comprises: In response to an interaction instruction sent by a user, determining a target mode based on a current position of the mobile device; If the target mode is the handle mode, then every first specified time period, the current position of the mobile device is obtained, the position of the screen cursor in the AR device is controlled based on the current position of the mobile device, and when a target determination instruction sent by the user is received, the screen cursor in the AR device is controlled by using the target determination instruction to complete the interactive operation; If the target mode is the touch mode, projection is performed through the micro-projector in the mobile device to obtain a target recording area and a non-recording area; if the interaction instruction is an input operation of the user in the recording area, the target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed in the display interface of the AR device; if the interaction instruction is an interaction instruction of the user in the non-recording area, the display interface in the AR device is controlled based on the interaction instruction.

2. The method according to claim 1, characterized in that The determining of the target mode based on the current position of the mobile device includes: Every second specified time period, obtaining the current position and posture of the mobile device; comparing the current pose with a previous pose of the mobile device; If the current posture is different from the previous posture, determining that the target mode of the mobile device is the handle mode; or, If the current posture is the same as the previous posture, determining whether the number of times the current posture of the mobile device is obtained is greater than a specified number; If not, return to obtaining the current position and posture of the mobile device every second specified time period until it is determined that the current position and posture are different from the previous position and posture or the number of times the current position and posture of the mobile device are obtained is greater than a specified number; or, If so, it is determined that the target model of the mobile device is the touch mode.

3. The method according to claim 1, characterized in that The projecting by the micro projector in the mobile device to obtain the target recording area and the non-recording area includes: Projecting by means of a micro-projector in the mobile device to obtain a projection area; In response to a user's selection operation on each recording area preset in the projection area, obtaining the target recording area; The other projection areas in the projection area except the target recording area are determined as the non-recording areas.

4. The method according to claim 1, characterized in that The step of determining a target sliding track in response to the input operation includes: When the user's finger begins to touch the recording area, a first distance of a specified number is acquired through a first sensor of the AR handle, wherein the first distance is a distance between the first sensor and the user's finger; and Acquire, by means of a second sensor of the AR handle, a second distance of a specified number, wherein the second distance is a distance between the second sensor and a finger of the user; Obtaining a specified number of distance sets according to the specified number of first distances and the specified number of second distances, wherein any one of the distance sets includes the first distance and the second distance; Based on the specified number of distance sets, get the radius of the finger touch area; At every third specified time period, the first distance and the second distance are acquired, and according to the radius of the finger touch area, the first distance and the second distance, the current position coordinates of the user's finger on the display interface of the AR device are obtained; According to each current position coordinate, the target sliding trajectory is obtained.

5. The method according to claim 4, characterized in that The step of obtaining the radius of the finger touch area based on a specified number of distance sets includes: For any distance set, according to the first distance and the second distance in the distance set, the middle center of the finger touch area corresponding to the distance set is obtained; and Based on the ordinate of the middle circle center of the finger touch area and the target distance, obtaining the middle radius of the finger touch area corresponding to the distance set, wherein the target distance is the distance between the first sensor and the second sensor; The radius of the finger touch area is obtained according to the middle radius of the finger touch area corresponding to each distance set.

6. The method according to claim 5, characterized in that The obtaining, according to the first distance and the second distance in the distance set, the middle center of the finger touch area corresponding to the distance set includes: The abscissa of the center of the middle circle of the finger touch area corresponding to the distance set is obtained by the following formula: Among them, x n is the horizontal coordinate of the center of the middle circle of the finger touch area corresponding to the distance set n, is the first distance corresponding to the distance set n, is a second distance corresponding to the distance set n, D1 is a distance between the first sensor and a preset coordinate origin, and D2 is a preset distance between the first sensor and the second sensor; The ordinate of the center of the middle circle of the finger touch area corresponding to the distance set is obtained by the following formula: Among them, y n is the ordinate of the middle circle center of the finger touch area corresponding to the distance set n, and l is the first distance or the second distance; The obtaining, based on the ordinate of the middle circle center of the finger touch area and the target distance, the middle radius of the finger touch area corresponding to the distance set includes: The middle radius of the finger touch area corresponding to the distance set is obtained by the following formula: Among them, r n is the middle radius of the finger touch area corresponding to the distance set n.

7. The method according to claim 5, characterized in that The obtaining the radius of the finger touch area according to the middle radius of the finger touch area corresponding to each distance set includes: The average value of the middle radii of the finger touch areas corresponding to the distance sets is determined as the radius of the finger touch area.

8. The method according to claim 4, characterized in that The obtaining, according to the radius of the finger touch area, the first distance, and the second distance, the current position coordinates of the user's finger on the display interface of the AR device includes: The current horizontal position coordinates of the user's finger on the display interface of the AR device are obtained by the following formula: Wherein, Xm is the mth current horizontal position coordinate of the user's finger in the display interface of the AR device, is the mth first distance, is the mth second distance, D1 is the distance between the first sensor and a preset coordinate origin, D2 is the preset distance between the first sensor and the second sensor, and N is a preset value; The current vertical position coordinate of the user's finger on the display interface of the AR device is obtained by the following formula: Among them, Y m is the mth current vertical position coordinate of the user's finger on the display interface of the AR device, and R is the radius of the finger touch area.

9. A mobile device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: In response to an interaction instruction sent by a user, determining a target mode based on a current position of the mobile device; If the target mode is the handle mode, then every first specified time period, the current position of the mobile device is obtained, the position of the screen cursor in the AR device is controlled based on the current position of the mobile device, and when a target determination instruction sent by the user is received, the screen cursor in the AR device is controlled by using the target determination instruction to complete the interactive operation; If the target mode is the touch mode, projection is performed through the micro-projector in the mobile device to obtain a target recording area and a non-recording area; if the interaction instruction is an input operation of the user in the recording area, the target sliding trajectory is determined in response to the input operation, and the target sliding trajectory is superimposed and displayed in the display interface of the AR device; if the interaction instruction is an interaction instruction of the user in the non-recording area, the display interface in the AR device is controlled based on the interaction instruction.

10. The mobile device according to claim 9, characterized in that The processor executes the projection through the micro projector in the mobile device to obtain the target recording area and non-recording area, and is specifically configured as follows: Projecting by means of a micro-projector in the mobile device to obtain a projection area; In response to a user's selection operation on each recording area preset in the projection area, obtaining the target recording area; The other projection areas in the projection area except the target recording area are determined as the non-recording areas.

Images