AR glasses display terminal device key input synchronous response control system and method
The AR glasses display terminal device key input synchronization response control system is used to accurately identify the feedback coordinate points of the fingertips and perform synchronization control, which solves the incoordination of operation in the interaction between AR glasses and terminal devices and the operation and terminal devices. The problem of misjudgment improves the accuracy and coordination of user operations.
Patent Information
- Application Number
- CN202410464001.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-04-17
AI Technical Summary
When interacting with the terminal device using AR glasses, since the control area is displayed on the terminal device rather than on the AR glasses, the user's operation is inconsistent, and the structured optical feedback data of the optical module is easily misjudged by the interfering object, causing the game Operation error.
Design a synchronization response control system for key input of AR glasses display terminal equipment, obtain structured optical feedback data of terminal equipment through optical modules, and filter out effective feedback coordinate points at fingertips in combination with DBSCAN density clustering method, set virtual keys and perform synchronization Control to ensure accurate identification and operation of feedback coordinate points at fingertips.
It improves the coordination and accuracy of user operations, reduces game operation errors caused by misjudgment of interfering objects, and enhances the synchronization control effect between AR glasses and terminal devices.
Smart Images

Figure CN118312064B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-computer interaction, and in particular to a control system and method for synchronous response to key input of an AR glasses display terminal device. Background Art
[0002] When playing games on handheld or mobile devices (such as Honor of Kings), the traditional gesture game will cause the fingertips to block the game. After the emergence of AR glasses, the game screen on the handheld or mobile device is synchronized to the AR glasses, and the game screen is displayed through the AR glasses. The game is controlled by the fingertips on the handheld or mobile device, which improves the technical problem of fingertips blocking the game screen.
[0003] However, the study found that when AR glasses interact with terminal devices, the corresponding control area cannot be seen on the AR glasses, but the control area is on the terminal device, which causes incoordination of control methods and poor control induction for users;
[0004] Technicians introduced optical modules, and by mounting the optical modules on mobile phone cases, or installing them separately on handheld or mobile devices, when performing synchronous button operations of AR glasses by obtaining structured light feedback data from the optical modules, feedback coordinate points of flying insects, hair, and other fingers will appear, affecting the judgment of the AR glasses and causing errors in game operations. Summary of the invention
[0005] The purpose of the present invention is to provide an AR glasses display terminal device key input synchronous response control system and method, which solves the above-mentioned technical problems pointed out in the prior art.
[0006] The present invention provides an AR glasses display terminal device key input synchronous response control system, including AR glasses, a shell, and a terminal device;
[0007] The AR glasses are connected to the housing and the terminal device respectively; the housing is mounted on the terminal device; the housing includes an optical module and a housing body;
[0008] The terminal device includes a screen recording module, a button module, and a synchronization module;
[0009] The AR glasses include a display module and a virtual button module;
[0010] The optical module is used to emit light above the terminal device and obtain structured light feedback data above the terminal device through the light;
[0011] The screen recording module on the above-mentioned terminal device is used to obtain the screen of the terminal device and send the screen of the terminal device to the display module on the AR glasses;
[0012] The button module on the terminal device is used to obtain the position of the real button in the terminal device screen, and send the position of the real button in the terminal device screen to the virtual button module of the AR glasses;
[0013] The virtual key module on the AR glasses is used to set a virtual key at a corresponding position of the AR glasses according to the position of the real key in the terminal device screen sent by the key module;
[0014] The display module on the AR glasses is used to obtain the screen of the terminal device sent by the screen recording module, and synchronize the screen of the terminal device to the AR glasses;
[0015] The synchronization module on the terminal device is used to obtain the structured light feedback data in the optical module, and obtain the feedback coordinate point of the fingertip according to the structured light feedback data, and when the vertical distance from the feedback coordinate point of the fingertip to the terminal device is less than or equal to the preset shortest distance threshold s between the lowest point of the fingertip and the terminal device, a semi-transparent virtual button is synchronously displayed on the AR glasses;
[0016] The synchronization module on the above-mentioned terminal device is also used to pre-set touch points on virtual keys on the AR glasses corresponding to real keys of the terminal device; and control the touch points to operate synchronously with the real keys of the terminal device.
[0017] Preferably, the synchronization module on the terminal device is specifically used to preset the shortest distance threshold s between the lowest point of the fingertip and the terminal device, obtain the coordinate point of the current lowest point of the fingertip through the structured light feedback data, determine the coordinate point of the current lowest point of the fingertip as the target point; and calculate the vertical distance from the target point to the terminal device;
[0018] Determine whether the vertical distance from the target point to the terminal device is less than or equal to a threshold s of the shortest distance between the lowest point of the fingertip and the terminal device;
[0019] If the distance from the target point to the terminal device is less than or equal to the shortest distance threshold s between the lowest point of the fingertip and the terminal device, further determine whether the area below the current target point is located at the location of the area of each real button in the screen of the terminal device;
[0020] If the area below the current target point is located at the area of each real button in the screen of the terminal device, the real button below the current target point is determined to be the target real button; and the virtual button corresponding to the target real button in the display screen of the AR glasses is displayed as a semi-transparent virtual button;
[0021] If the distance from the target point to the terminal device is 0, the touch point is displayed at a position corresponding to the position of the target point in the display screen of the AR glasses, the touch point is synchronized with the position of the target point, and corresponding synchronization control operations are performed.
[0022] Preferably, in a specific technical solution, the synchronization module is specifically used to obtain a plurality of feedback coordinate points through the structured light feedback data, and obtain a plurality of valid feedback coordinate points from the plurality of feedback coordinate points by a preset time period t and a preset shortest residence time period threshold a of the feedback coordinate point above the terminal device within the time period t;
[0023] Using the DBSCAN density clustering method to cluster the multiple valid feedback coordinate points, obtain multiple valid feedback coordinate point clusters, and determine the valid feedback coordinate point clusters as the point clusters to be selected;
[0024] A minimum threshold value L of the number of valid feedback coordinate points in the effective feedback coordinate point cluster of the fingertip is preset, and if the number of valid feedback coordinate points in the to-be-selected point cluster is greater than or equal to the minimum threshold value L of the number of valid feedback coordinate points in the effective feedback coordinate point cluster of the fingertip, the to-be-selected point cluster is determined to be a to-be-selected target point cluster; and the to-be-selected target point cluster with the largest number of valid feedback coordinate points in the to-be-selected target point cluster is selected as the target point cluster;
[0025] The vertical distances from all the valid feedback coordinate points to the terminal device are calculated respectively; and the valid feedback coordinate point in the target point cluster that is closest to the terminal device in vertical distance is determined as the target point.
[0026] Accordingly, the present invention also proposes a method for controlling a synchronous response to a key input of an AR glasses display terminal device, comprising the following steps:
[0027] The AR glasses establish a connection with the terminal device, and synchronize the screen of the terminal device to the AR glasses to obtain the AR glasses display screen; the AR glasses establish a connection with the optical module, and obtain the structured light feedback data of the optical module in real time;
[0028] The AR glasses obtain the positions of the regions of each real key in the screen of the terminal device, and set virtual keys at positions corresponding to the positions of the regions of each real key in the screen of the terminal device in the screen of the AR glasses; and preset touch points in the regions where each virtual key is located;
[0029] The optical module is initialized, a shortest distance threshold s between the lowest point of the fingertip and the terminal device is preset, the coordinate point of the current lowest point of the fingertip is obtained through the structured light feedback data, and the coordinate point of the current lowest point of the fingertip is determined as the target point; and the vertical distance from the target point to the terminal device is obtained;
[0030] Determine whether the distance from the target point to the terminal device is less than or equal to a threshold s of the shortest distance between the lowest point of the fingertip and the terminal device;
[0031] If the distance from the target point to the terminal device is less than or equal to the shortest distance threshold s between the lowest point of the fingertip and the terminal device, further determine whether the area below the current target point is located at the location of the area of each real button in the screen of the terminal device;
[0032] If the area below the current target point is located at the area of each real button in the screen of the terminal device, the real button below the current target point is determined to be the target real button; and the virtual button corresponding to the target real button in the display screen of the AR glasses is displayed as a semi-transparent virtual button;
[0033] If the vertical distance from the target point to the terminal device is 0, the touch point is displayed at a position corresponding to the position of the target point in the display screen of the AR glasses, the touch point is synchronized with the position of the target point, and corresponding synchronization control operations are performed.
[0034] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:
[0035] It can be seen from the analysis of the above-mentioned AR glasses display terminal device key input synchronization response control system and method provided by the present invention that in specific application, the synchronization module synchronizes the screen of the terminal device to the AR glasses to obtain the AR glasses display screen;
[0036] A virtual button corresponding to a real button in the terminal device screen is set in the AR glasses display screen; at the same time, the AR glasses obtain structured light feedback data of the optical module, and obtain feedback coordinate points of the object above the terminal device according to the structured light feedback data; according to a preset time period t and a threshold value a of the shortest residence time period of the preset feedback coordinate point above the terminal device within the time period t, a plurality of valid feedback coordinate points are screened out from the plurality of feedback coordinate points;
[0037] Clustering multiple valid feedback coordinate points by DBSCAN density clustering method to obtain multiple valid feedback coordinate point clusters; based on the preset minimum valid feedback coordinate point number threshold in the valid feedback coordinate point cluster of the fingertip, the valid feedback coordinate point clusters of other interference objects are screened out, and finally the feedback coordinate point cluster with the largest number in the feedback coordinate point cluster is determined as the target point cluster (that is, the valid feedback coordinate point cluster of the finger);
[0038] A valid feedback coordinate point closest to the terminal device is selected from the target point cluster as the target point (that is, the lowest point of the fingertip is the target point); at the same moment when the target point touches the terminal device, the virtual key on the AR glasses corresponding to the real key of the terminal device touched by the target point is displayed as the largest, and the position of the terminal device touched by the target point corresponds to the position on the AR glasses where a pre-set touch point is displayed (that is, the target point touches the current real key, and the touch point appears on the virtual key of the AR glasses corresponding to the current real key); finally, when controlling the game, the virtual key on the AR glasses controlled by the touch point is synchronized with the real key on the terminal device controlled by the target point, thereby performing synchronous control operations. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 This is a schematic diagram of a synchronous response control system for key input of an AR glasses display terminal device provided in the first embodiment of the present invention;
[0041] Figure 2 A schematic diagram of the physical effect of the key input synchronization response control system of the AR glasses display terminal device provided in the first embodiment of the present invention;
[0042] Figure 3 A schematic diagram of the physical effect of the key input synchronization response control system of the AR glasses display terminal device provided in the first embodiment of the present invention;
[0043] Figure 4 A schematic diagram of the overall operation flow of a method for controlling a synchronous response to key input of an AR glasses display terminal device provided in the second embodiment of the present invention;
[0044] Figure 5 A schematic diagram of the main operation flow of a method for controlling synchronous response to key input of an AR glasses display terminal device provided in Embodiment 3 of the present invention;
[0045] Figure 6 A schematic diagram of the operation flow of the first case of obtaining a target point in a method for synchronous response control of key input of an AR glasses display terminal device provided in Embodiment 3 of the present invention;
[0046] Figure 7 A schematic diagram of the operation flow of the second case of obtaining a target point in a method for synchronous response control of key input of an AR glasses display terminal device provided in Embodiment 3 of the present invention;
[0047] Figure 8 This is a schematic diagram of the operation flow of obtaining a cluster of points to be selected in a method for synchronous response control of key input of an AR glasses display terminal device provided in Embodiment 3 of the present invention.
[0048] Labels: AR glasses 10; housing 20; terminal device 30; optical module 21; screen recording module 31; button module 32; display module 11; virtual button module 12; synchronization module 33. DETAILED DESCRIPTION
[0049] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] The present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings.
[0051] Embodiment 1
[0052] like Figure 1-3 As shown, the present invention proposes an AR glasses display terminal device key input synchronous response control system, including AR glasses 10, a housing 20, and a terminal device 30;
[0053] The above-mentioned Figure 1 This is the schematic diagram of the AR glasses display terminal device key input synchronous response control system; see also the attached Figure 2 And attached Figure 3 , attached Figure 2 And attached Figure 3 It is a schematic diagram of the physical effect of the key input synchronization response control system of the AR glasses display terminal device;
[0054] The AR glasses 10 are connected to the housing 20 and the terminal device 30 respectively; the housing 20 is installed on the terminal device 30 (the terminal device can generally be a mobile phone or a handheld game console); the housing 20 includes an optical module 21 and a housing body; that is, the structure of the housing body is equivalent to a "mobile phone case", but this "mobile phone case" is a housing structure in which the optical module 21 is installed on the top side of the housing;
[0055] The terminal device 30 includes a screen recording module 31, a key module 32, and a synchronization module 33; the terminal device can generally be a mobile phone or a handheld game console, and the functions of the screen recording module 31 and the key module 32 are realized at the same time, which is actually realized by an application installed on the terminal device;
[0056] The AR glasses 10 include a display module 11 and a virtual key module 12; that is, the AR glasses display terminal device key input synchronous response control system provided in the embodiment of the present invention involves three aspects of adaptive improvement technologies. One aspect is to cooperate with the installation of a shell with an optical module (of course, the shell can also be omitted, and the optical module 21 can be directly bonded as an accessory to one side or multiple sides of the terminal device 30, and connected to the interface on the terminal device 30 through a data cable to realize power supply to the optical module); on the other hand, the terminal device 30 also needs to adaptively install the function programs of the screen recording module 31, the key module 32, and the synchronization module 33, which are functions generated by installing the APP program; at the same time, as the main system in the embodiment of the present application, the AR glasses 10 also need to install the application program that realizes the functions of the "display module 11, virtual key module 12";
[0057] The optical module 21 is used to emit light above the terminal device and obtain structured light feedback data above the terminal device through the light (that is, the optical module 21 is actually installed at the side of the terminal device to emit light above the terminal device and obtain structured light feedback data above the terminal device);
[0058] The screen recording module 31 on the terminal device 30 is used to obtain the screen of the terminal device and send the screen of the terminal device to the display module 11 on the AR glasses;
[0059] The button module 32 on the terminal device 30 is used to obtain the position of the real button in the terminal device screen, and send the position of the real button in the terminal device screen to the virtual button module 12 of the AR glasses;
[0060] The virtual key module 12 on the AR glasses 10 is used to set a virtual key at a corresponding position of the AR glasses according to the position of the real key in the terminal device screen sent by the key module 32;
[0061] The display module 11 on the AR glasses 10 is used to obtain the screen of the terminal device sent by the screen recording module 31, and synchronize the screen of the terminal device to the AR glasses;
[0062] The synchronization module 33 on the terminal device 30 is used to obtain the structured light feedback data in the optical module, and obtain the feedback coordinate point of the fingertip according to the structured light feedback data, and when the vertical distance from the feedback coordinate point of the fingertip to the terminal device is less than or equal to the preset shortest distance threshold s between the lowest point of the fingertip and the terminal device, a semi-transparent virtual key is synchronously displayed on the AR glasses (the above virtual key is a semi-transparent virtual key in a picture-in-picture form for synchronous display); and the closer the distance between the feedback coordinate point of the fingertip and the terminal device is, the larger the semi-transparent virtual key synchronously displayed on the AR glasses is;
[0063] The synchronization module 33 on the above-mentioned terminal device 30 is also used to pre-set touch points on virtual keys on the AR glasses corresponding to real keys of the terminal device; and control the touch points to perform synchronous operations with the real keys of the terminal device.
[0064] The synchronization module on the terminal device 30 is specifically used to preset the shortest distance threshold s between the lowest point of the fingertip and the terminal device, obtain the coordinate point of the current lowest point of the fingertip through the structured light feedback data (the coordinate point of the lowest point of the fingertip is the feedback coordinate point with the shortest vertical distance from the feedback coordinate point of the fingertip to the terminal device), determine the coordinate point of the current lowest point of the fingertip as the target point; and calculate the vertical distance from the target point to the terminal device;
[0065] Determine whether the vertical distance from the target point to the terminal device is less than or equal to a threshold s of the shortest distance between the lowest point of the fingertip and the terminal device;
[0066] If the distance from the target point to the terminal device is less than or equal to the shortest distance threshold s between the lowest point of the fingertip and the terminal device, further determine whether the area below the current target point is located at the location of the area of each real button in the screen of the terminal device;
[0067] If the area below the current target point is located at the area of each real button in the screen of the terminal device, the real button below the current target point is determined to be the target real button; and the virtual button corresponding to the target real button in the display screen of the AR glasses is displayed as a semi-transparent virtual button;
[0068] If the vertical distance from the target point to the terminal device is 0, the touch point is displayed at a position corresponding to the position of the target point in the display screen of the AR glasses, the touch point is synchronized with the position of the target point, and corresponding synchronization control operations are performed.
[0069] In a specific technical solution, the synchronization module is specifically used to obtain a plurality of feedback coordinate points through the structured light feedback data, and obtain a plurality of valid feedback coordinate points from the plurality of feedback coordinate points through a preset time period t and a preset shortest residence time period threshold a of the feedback coordinate point above the terminal device within the time period t;
[0070] Using the DBSCAN density clustering method to cluster the multiple valid feedback coordinate points, obtain multiple valid feedback coordinate point clusters, and determine the valid feedback coordinate point clusters as the point clusters to be selected;
[0071] A minimum threshold value L of the number of valid feedback coordinate points in the effective feedback coordinate point cluster of the fingertip is preset, and if the number of valid feedback coordinate points in the to-be-selected point cluster is greater than or equal to the minimum threshold value L of the number of valid feedback coordinate points in the effective feedback coordinate point cluster of the fingertip, the to-be-selected point cluster is determined to be a to-be-selected target point cluster; and the to-be-selected target point cluster with the largest number of valid feedback coordinate points in the to-be-selected target point cluster is selected as the target point cluster;
[0072] The vertical distances from all the valid feedback coordinate points to the terminal device are calculated respectively; and the valid feedback coordinate point in the target point cluster that is the shortest in vertical distance to the terminal device is determined as the target point.
[0073] In summary, in the above-mentioned AR glasses display terminal device key input synchronization response control system provided by the present invention, during specific operation, the synchronization module synchronizes the screen of the terminal device to the AR glasses to obtain the AR glasses display screen;
[0074] A virtual button corresponding to a real button in the terminal device screen is set in the AR glasses display screen; at the same time, the AR glasses obtain structured light feedback data of the optical module, and obtain feedback coordinate points of the object above the terminal device according to the structured light feedback data; according to a preset time period t and a threshold value a of the shortest residence time period of the preset feedback coordinate point above the terminal device within the time period t, a plurality of valid feedback coordinate points are screened out from the plurality of feedback coordinate points;
[0075] Clustering multiple valid feedback coordinate points by DBSCAN density clustering method to obtain multiple valid feedback coordinate point clusters; based on the preset minimum valid feedback coordinate point number threshold in the valid feedback coordinate point cluster of the fingertip, the valid feedback coordinate point clusters of other interference objects are screened out, and finally the feedback coordinate point cluster with the largest number in the feedback coordinate point cluster is determined as the target point cluster (that is, the valid feedback coordinate point cluster of the finger);
[0076] A valid feedback coordinate point closest to the terminal device is selected from the target point cluster as the target point (that is, the lowest point of the fingertip is the target point); at the same moment when the target point touches the terminal device, the virtual key on the AR glasses corresponding to the real key of the terminal device touched by the target point is displayed as the largest, and the position of the terminal device touched by the target point corresponds to the position on the AR glasses where a pre-set touch point is displayed (that is, the target point touches the current real key, and the touch point appears on the virtual key of the AR glasses corresponding to the current real key); finally, when controlling the game, the virtual key on the AR glasses controlled by the touch point is synchronized with the real key on the terminal device controlled by the target point, thereby performing synchronous control operations.
[0077] Embodiment 2
[0078] like Figure 4 As shown, the present invention also proposes a method for controlling a key input synchronization response of an AR glasses display terminal device, comprising the following steps:
[0079] Step S1: Initialize the connection relationship and obtain structured light feedback data;
[0080] Step S2: Acquire an AR glasses display screen; set virtual keys corresponding to real keys on the terminal device display screen and touch points corresponding to the virtual keys on the AR glasses display screen;
[0081] Step S3: presetting a shortest distance threshold s, and acquiring a plurality of feedback coordinate points according to the structured light feedback data;
[0082] Step S4: Preset the time period t and the shortest residence time period threshold a of the feedback coordinate point above the terminal device within the time period t, obtain the residence time a' of the feedback coordinate point within the time period t, and select multiple valid feedback coordinate points from the multiple feedback coordinate points;
[0083] Step S5: clustering the multiple valid feedback coordinate points using the DBSCAN density clustering method to obtain multiple candidate point clusters;
[0084] Step S6: Preset a minimum number threshold L of effective feedback coordinate points in the effective feedback coordinate point cluster of the fingertip, and obtain a target point cluster to be selected; and select a target point cluster with the number of effective feedback coordinate points from the target point cluster to be selected as the target point cluster;
[0085] Step S7: Calculate the vertical distances from all the valid feedback coordinate points to the terminal device respectively; select a feedback coordinate point in the target point cluster with the shortest vertical distance from the terminal device as the target point, and obtain the vertical distance from the target point to the terminal device;
[0086] Step S8: determining whether the vertical distance from the target point to the terminal device is less than or equal to the shortest distance threshold s; if the distance from the target point to the terminal device is less than or equal to the shortest distance threshold s, further determining whether there is a real button below the target point; if so, controlling the virtual button corresponding to the current real button on the AR glasses to be semi-transparently displayed;
[0087] Step S9: Determine whether the vertical distance from the target point to the terminal device is 0. If so, display the touch point in the virtual key on the AR glasses corresponding to the real key where the target point is located; the touch point is synchronized with the target point.
[0088] Embodiment 3
[0089] like Figure 5 As shown, accordingly, the present invention also proposes a method for controlling a key input synchronization response of an AR glasses display terminal device, comprising the following steps: the method for controlling a key input synchronization response of an AR glasses display terminal device adopted in the above-mentioned embodiment 3 is actually the same as the method in embodiment 2, except that the control method in embodiment 3 of the present invention is a specific detailed processing scheme of the control method in embodiment 2;
[0090] Step S10: the AR glasses establish a connection with the terminal device (handheld device or mobile device), and synchronize the screen of the terminal device to the AR glasses to obtain the AR glasses display screen; the AR glasses establish a connection with the optical module, and obtain the structured light feedback data of the optical module in real time;
[0091] The terminal device is usually a handheld game console or a mobile phone device; the optical module is installed on the terminal device as an external connection device and connected to the AR glasses;
[0092] The structured light feedback data of the optical module refers to the optical module emitting light above the terminal device, and obtaining the coordinate point of the object above the terminal device (usually the object specifically refers to the fingertips of the human body) and the distance from the coordinate point to the terminal device through light sensing; that is, the structured light feedback data includes the coordinate point of the object and the distance from the coordinate point to the terminal device;
[0093] Step S20: the AR glasses obtain the positions of the regions of the real buttons in the screen of the terminal device, and set virtual buttons at positions corresponding to the positions of the regions of the real buttons in the screen of the terminal device in the screen of the AR glasses; and preset touch points in the regions where the virtual buttons are located;
[0094] Step S30: the optical module is initialized, a threshold value s of the shortest distance between the lowest point of the fingertip and the terminal device is preset, the coordinate point of the current lowest point of the fingertip is obtained through the structured light feedback data (the coordinate point of the lowest point of the fingertip is the feedback coordinate point with the shortest vertical distance from the feedback coordinate point of the fingertip to the terminal device), and the coordinate point of the current lowest point of the fingertip is determined as the target point; and the vertical distance from the target point to the terminal device is obtained;
[0095] Determine whether the vertical distance from the target point to the terminal device is less than or equal to a threshold s of the shortest distance between the lowest point of the fingertip and the terminal device;
[0096] If the vertical distance from the target point to the terminal device is less than or equal to the shortest distance threshold s between the lowest point of the fingertip and the terminal device, further determine whether the area below the current target point is located at the location of the areas of each real button in the screen of the terminal device (there will be multiple real buttons in the screen of the terminal device, but the real buttons are not evenly distributed in the entire screen, so it is necessary to determine whether the area below the target point belongs to one of the multiple real buttons);
[0097] If the area below the current target point is located at the position of the areas of the various real buttons in the screen of the terminal device, the real button below the current target point is determined to be the target real button; (the position of the current real button is to determine the position of the real button below the current target point. For example, in the King of Glory game, usually, there will be an upper left map real button area, a lower left direction real button area, an upper right setting real button area, and a lower right skill real button area. In addition, in multiple real button areas, there will be smaller real buttons for players to operate. For example, in the lower left direction real button area, there will be an up walk real button, a down walk real button, a left walk real button, and a right walk real button. In the upper left map real button area, there will be an open map real button, a close map real button, an open voice real button, a close voice real button, a map mark real button, etc.; in an embodiment of the present invention, determining the position of the current real button is to determine the position of the current smallest real button), and the virtual button corresponding to the target real button in the display screen of the AR glasses is displayed as a translucent virtual button;
[0098] It should be noted that the virtual key corresponding to the target real key in the display screen of the AR glasses is displayed as a translucent virtual key, and is displayed as the smallest translucent virtual key when the vertical distance from the target point to the terminal device is equal to the shortest distance threshold s between the lowest point of the fingertip and the terminal device. As the distance from the target point to the terminal device is closer, the translucent virtual key is displayed larger (when the user's finger is suspended above the screen and the distance from the screen is less than the threshold, it is displayed as a circle; the farther the finger is, the larger the circle is; the closer the finger is, the smaller and deeper the circle is), until the target point touches the current real key of the terminal device at the same moment, the translucent virtual key is displayed as the largest; and as the distance from the target point to the terminal device is farther, the translucent virtual key is displayed smaller, until the distance from the target point to the terminal device is greater than the shortest distance threshold s between the lowest point of the fingertip and the terminal device, the translucent virtual key is hidden.
[0099] Step S40: If the vertical distance from the target point to the terminal device is 0, the touch point is displayed at the position corresponding to the position of the target point in the display screen of the AR glasses (the "displaying the touch point at the position corresponding to the position of the target point in the display screen of the AR glasses" means that in the display screen of the AR glasses, a pre-set touch point is displayed at the position of the virtual key corresponding to the real key of the mobile phone touched by the fingertip), the touch point is synchronized with the position of the target point, and corresponding synchronization control operations are performed.
[0100] Specifically, Figure 6As shown, in step S30, the coordinate point of the lowest point of the current fingertip is obtained through the structured light feedback data, and the coordinate point of the lowest point of the current fingertip is determined as the target point, including the following operation steps:
[0101] Step S31: obtaining a plurality of feedback coordinate points through the structured light feedback data, and obtaining a plurality of valid feedback coordinate points from the plurality of feedback coordinate points (the feedback coordinate points include user fingertip feedback coordinate points, hair feedback coordinate points, and flying insect feedback coordinate points);
[0102] It should be noted that when the distance from the fingertip to the terminal device is less than the shortest distance threshold s between the lowest point of the fingertip and the terminal device, the structured light feedback data obtained by the optical module will include the feedback coordinate points of the fingertips, the feedback coordinate points of fallen hair, the feedback coordinate points of flying insects, and the feedback coordinate points of other people's fingers (but under normal circumstances, the feedback coordinate points of other people's fingers often flash by and will not affect the current user's game operations). At this time, it is necessary to screen out the relatively fixed feedback coordinate points above the terminal device, that is, the feedback coordinate points of objects that have been above the terminal device for a long period of time.
[0103] Step S32: clustering the multiple valid feedback coordinate points using the DBSCAN density clustering method to obtain multiple valid feedback coordinate point clusters, and determining the valid feedback coordinate point clusters as the point clusters to be selected;
[0104] It should be noted that the DBSCAN density clustering method is a conventional density clustering method. Because the range of the point cluster to be selected is very large, there may be valid feedback coordinate points of the user's hair, fingertips, and flying insects inside. Since there will be a clearance distance between the hair, fingertips, and flying insects, that is to say, the current valid feedback coordinate point of the fingertip will inevitably be spatially isolated from the valid feedback coordinate point of the hair or flying insect. After giving a distance threshold, the DBSCAN density clustering method can directly cluster the surrounding valid feedback coordinate points to obtain multiple valid feedback coordinate point clusters.
[0105] The effective feedback coordinate point clusters are multiple effective feedback coordinate points obtained by the optical module according to the feedback light data of the hair, fingertips, and flying insects above the terminal device, and the effective feedback coordinate points are clustered by DBSCAN density method to obtain multiple effective feedback coordinate point clusters, including effective feedback coordinate point clusters of hair, effective feedback coordinate point clusters of fingertips, and effective feedback coordinate point clusters of flying insects;
[0106] Step S33: Preset a minimum threshold value L for the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip; if the number of valid feedback coordinate points in the to-be-selected point cluster is greater than or equal to the minimum threshold value L for the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip, determine the to-be-selected point cluster as the to-be-selected target point cluster (the target point cluster is the valid feedback coordinate point cluster of the fingertip); and select the to-be-selected target point cluster with the largest number of valid feedback coordinate points in the to-be-selected target point cluster as the target point cluster;
[0107] It should be noted that if the number of valid feedback coordinate points in the to-be-selected point cluster is less than the minimum threshold value L of the number of valid feedback coordinate points in the fingertip valid feedback coordinate point cluster, it can be determined that there is only hair or flying insects above the current mobile phone screen, and no fingertips are above the mobile phone screen, then the next step of processing will not be performed at this time.
[0108] Explanation: This operation step may also include presetting a minimum threshold value M of the number of valid feedback coordinate points in the hair valid feedback coordinate point cluster. If the number of valid feedback coordinate points in the to-be-selected point cluster is greater than or equal to the minimum threshold value M of the number of valid feedback coordinate points in the hair valid feedback coordinate point cluster, then the to-be-selected point cluster is determined to be a to-be-selected hair point cluster. In addition, the step of determining the to-be-selected flying insect point cluster is consistent with the above operation, and no further explanation is given for this.
[0109] When a user uses a terminal device such as a mobile phone to play games, the point cluster above the terminal device such as the mobile phone usually only has the fingertips as the point cluster with the largest number of coordinate points. Therefore, it is necessary to select a candidate target point cluster with the largest number of valid feedback coordinate points as the target point cluster.
[0110] Step S34: respectively calculating the vertical distances from all the valid feedback coordinate points to the terminal device; and determining the valid feedback coordinate point in the target point cluster that is the shortest in vertical distance to the terminal device as the target point.
[0111] During the specific implementation of the technical solution adopted in the embodiment of the present invention, the technicians found that the fingertips may have fallen hair and other debris on them, and the feedback coordinate points of these debris will cause errors in obtaining the target points; therefore, when screening and determining the effective feedback coordinate points, it is necessary to screen out the feedback coordinate points of the hair and other debris on the fingertips, and simulate and construct the effective feedback coordinate points of the fingertips blocked by the hair, so as to construct a complete cluster of effective feedback coordinate points of the fingertips, and then use the complete cluster of effective feedback coordinate points of the fingertips to screen and determine the target points.
[0112] Specifically, Figure 7 As shown, in step S30, the coordinate point of the lowest point of the current fingertip is obtained through the structured light feedback data, and the coordinate point of the lowest point of the current fingertip is determined as the target point, and the following operation steps are also included:
[0113] Step S35: obtaining a plurality of feedback coordinate points through the structured light feedback data, and obtaining a plurality of valid feedback coordinate points from the plurality of feedback coordinate points; performing clustering processing on the plurality of valid feedback coordinate points using the DBSCAN density clustering method to obtain a plurality of valid feedback coordinate point clusters, and determining the valid feedback coordinate point clusters as the point clusters to be selected;
[0114] Step S36: presetting a minimum threshold value L of the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip, and obtaining the number L' of valid feedback coordinate points in all the to-be-selected point clusters; if the number L' of valid feedback coordinate points in the to-be-selected point cluster is greater than or equal to the minimum threshold value L of the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip, then determining the to-be-selected point cluster as the to-be-selected target point cluster; and selecting the to-be-selected target point cluster with the largest number of valid feedback coordinate points in the to-be-selected target point cluster as the target point cluster;
[0115] Step S37: acquiring all valid feedback coordinate points in the target point cluster, presetting a maximum 3D distance threshold y between two adjacent valid feedback coordinate points, and calculating the 3D distance between two adjacent valid feedback coordinate points; determining the magnitude relationship between the 3D distance between the two adjacent valid feedback coordinate points and the maximum 3D distance threshold y between the two adjacent valid feedback coordinate points; if the 3D distance between the two adjacent valid feedback coordinate points is greater than the maximum 3D distance threshold between the two adjacent valid feedback coordinate points, acquiring all valid feedback coordinate points whose 3D distance between two adjacent valid feedback coordinate points is greater than the maximum 3D distance threshold between the two adjacent valid feedback coordinate points, and then acquiring the 3D distances between other two adjacent valid feedback coordinate points, and calculating the average value of the 3D distances between the other two adjacent valid feedback coordinate points;
[0116] It should be noted that the two adjacent valid feedback coordinate points refer to two valid feedback coordinate points that are adjacent to each other among the valid feedback coordinate points;
[0117] The above calculation of the 3D distance between two adjacent valid feedback coordinate points refers to the distance between two adjacent valid feedback coordinate points in three-dimensional space; the 3D distance between two adjacent valid feedback coordinate points can be calculated using the Euclidean distance formula. Assuming that the two adjacent valid feedback coordinate points are P1(x1,y1,z1) and P2(x2,y2,z2), the distance d between them can be expressed by the following formula:
[0118]
[0119] This formula uses the Pythagorean theorem to obtain the straight-line distance between two points, that is, the length of the path required to reach another point from one point; the above-mentioned method of calculating the 3D distance between two adjacent valid feedback coordinate points is prior art and is not the main invention of the present invention, and the embodiments of the present invention will not be repeated.
[0120] Step S38: According to the average value of the 3D distances between the other two adjacent valid feedback coordinate points, simulate and construct the valid feedback coordinate points of the blocked fingertip to obtain a complete cluster of valid feedback coordinate points of the fingertip;
[0121] It should be noted that, in the technical solution adopted in the above-mentioned embodiment of the present invention, the effective feedback coordinate points of the occluded fingertips are simulated and constructed by screening out the effective feedback coordinate points whose 3D distance between all the above-mentioned two adjacent effective feedback coordinate points is greater than the maximum 3D distance threshold between the two adjacent effective feedback coordinate points, and then the average value of the 3D distance between other two adjacent effective feedback coordinate points is used to simulate and construct the effective feedback coordinate points of the occluded fingertips from the screened-out parts in sequence, and the simulated effective feedback coordinate points of the occluded fingertips and other effective feedback coordinate points are merged to obtain a complete cluster of effective feedback coordinate points of the fingertips.
[0122] Step S39: Calculate the vertical distances from all the valid feedback coordinate points to the terminal device respectively; determine the valid feedback coordinate point in the complete cluster of valid feedback coordinate points of the fingertip that is closest to the terminal device in vertical distance as the target point.
[0123] The technical solutions adopted in the above steps S31 to S34 are recognition processing steps in a normal scenario where there is no hair stuck to the fingertips.
[0124] The technical solution adopted in the above steps S35-S39 takes into account that in actual operation, there may be debris such as fallen hair on the fingertips. The scene at this time is a scene under a specific situation. Therefore, when obtaining the target point, it is necessary to further identify the effective feedback coordinate points of the fingertips that are blocked by the effective feedback coordinate points of the hair; among them, steps S35-S36 are the same as the aforementioned steps S31-S33.
[0125] The above scheme is explained as follows: if the 3D distance between two adjacent valid feedback coordinate points is greater than the maximum 3D distance threshold between the two adjacent valid feedback coordinate points, it is proved that the two valid feedback coordinate points whose 3D distance between the two adjacent valid feedback coordinate points is greater than the maximum 3D distance threshold between the two adjacent valid feedback coordinate points are respectively the valid feedback coordinate points of the hair and the valid feedback coordinate points of the fingertips. In this case, it is necessary to filter out the valid feedback coordinate points of the hair and obtain the valid feedback coordinate point of the lowest point of the fingertips.
[0126] In the structured light feedback data, it is recognized that the valid feedback coordinate points belonging to the same fingertip are very evenly distributed. Therefore, for the same fingertip, the 3D distances between two adjacent valid feedback coordinate points in the effective feedback coordinate point cluster of the fingertip are roughly the same. Based on this conclusion, the valid feedback coordinate points containing hair in the entire target point cluster can be determined based on multiple coordinate points whose 3D distance between two adjacent valid feedback coordinate points is greater than the maximum 3D distance threshold between two adjacent valid feedback coordinate points, and the valid feedback coordinate points of hair can be screened out based on the average value of the 3D distance between two adjacent valid feedback coordinate points.
[0127] Then, the effective feedback coordinate points of the fingertips blocked by the hair are simulated and constructed according to the average value of the 3D distance between two adjacent effective feedback coordinate points, and finally a complete fingertip point cluster is obtained;
[0128] Specifically, in step S31 or step S35, a plurality of feedback coordinate points are obtained through the structured light feedback data, and a plurality of valid feedback coordinate points are obtained from the plurality of feedback coordinate points, including the following operation steps:
[0129] Step S311: randomly selecting any one of the multiple feedback coordinate points as the first feedback coordinate point; presetting a time period t, and presetting a threshold value a of the shortest stay time period of the feedback coordinate point above the terminal device within the time period t, and obtaining a time a' that the current first feedback coordinate point stays above the terminal device within the time period t; if the time a' that the current first feedback coordinate point stays above the terminal device within the time period t is greater than the threshold value a of the shortest stay time period of the feedback coordinate point above the terminal device within the time period t, then determining that the current first feedback coordinate point is a valid feedback coordinate point;
[0130] It should be noted that, in the embodiment of the present invention, the preset time period t is a relatively long time period, such as 1s or 2s; the shortest residence time period threshold a of the feedback coordinate point above the terminal device within the time period t is a relatively short time period, which refers to the shortest residence time period threshold of the feedback coordinate point above the mobile phone screen, which is usually 8300μs;
[0131] When determining that the current first feedback coordinate point is a valid feedback coordinate point, a≤a'≤t must be satisfied, that is, the time a' that the current first feedback coordinate point stays above the terminal device within the time period t must be greater than or equal to the shortest stay time period threshold a of the feedback coordinate point above the terminal device within the time period t, and at the same time, the time a' that the current first feedback coordinate point stays above the terminal device within the time period t is less than or equal to the time period t, so that the feedback coordinate points of objects that are more obvious above the mobile phone screen can be screened out.
[0132] Specifically, as Figure 8 shown, in step S32 or step S35, the DBSCAN density clustering method is used to cluster the multiple effective feedback coordinate points to obtain multiple effective feedback coordinate point clusters, and determine the effective feedback coordinate point clusters as candidate point clusters, including the following operation steps:
[0133] Step S321: Establish a sample set D, and input the multiple effective feedback coordinate points into the sample set D; preset a sample neighborhood distance threshold ∈; preset a minimum threshold MinPts for the number of samples in the sample neighborhood distance threshold ∈;
[0134] The ∈ is a constant; the MinPts is a constant
[0135] It should be noted that the sample neighborhood distance threshold ∈ refers to taking an effective feedback coordinate point as the center of a circle, and obtaining all effective feedback coordinate points whose distance from the current effective feedback coordinate point is less than or equal to the sample neighborhood distance threshold ∈; the minimum threshold MinPts for the number of samples in the sample neighborhood distance threshold ∈ refers to the minimum threshold for the number of all effective feedback coordinate points whose distance from the current effective feedback coordinate point is less than or equal to the sample neighborhood distance threshold ∈ when taking an effective feedback coordinate point as the center of a circle.
[0136] Step S322: Initialize the unvisited sample set P (the initialization of the unvisited sample set P refers to all effective feedback coordinate points in the sample set D);
[0137] Step S323: Randomly select any feedback coordinate point in the unvisited sample set P as the core point to be determined;
[0138] Step S324: According to the sample neighborhood distance threshold ∈, obtain the neighborhood sub-sample set of the core point to be determined, and obtain the number of samples in the neighborhood sub-sample set of the core point to be determined; if the number of samples in the neighborhood sub-sample set of the core point to be determined is greater than or equal to the minimum threshold MinPts for the number of samples in the sample neighborhood distance threshold ∈, then determine the core point to be determined as a core point, and determine the neighborhood sub-sample set of the core point to be determined as the neighborhood sub-sample set of the core point;
[0139] It should be noted that if the number of samples in the neighborhood sub-sample set of the core point to be determined < MinPts, then it can be determined that the current core point to be determined is a noise point or a core point in other candidate point clusters, and then it is screened out or temporarily placed.
[0140] Step S325: Establish a sub-sample set N, and put the core point and the neighborhood sub-sample set of the core point into the sub-sample set N, and determine the set sub-sample N as the candidate point cluster N;
[0141] Step S326: Obtain a new unvisited sample set PN, and repeat the above steps S323 to S325 until the unvisited sample set P=0, and obtain multiple point clusters to be selected.
[0142] Explanation: The technical solution adopted in the above embodiment of the present invention is to first establish a sample set D, and input the above-obtained multiple valid feedback coordinate points into the sample set D; preset the sample neighborhood distance threshold ∈ and preset the minimum threshold value MinPts of the number of samples in the sample neighborhood distance threshold ∈, and in subsequent operations, use the sample neighborhood distance threshold ∈ and the minimum threshold value MinPts of the number of samples in the sample neighborhood distance threshold ∈ to cluster the sample data in the sample set D (the sample data is the multiple valid feedback coordinate points) and obtain multiple clusters;
[0143] Initialize the unvisited sample set P. The sample set P is initialized to be exactly the same as the sample set D. After obtaining a cluster, it is necessary to filter out all valid feedback coordinate points contained in the clustered cluster from the unvisited sample set P.
[0144] Select any valid feedback coordinate point in the unvisited sample set P at any time as the core point to be determined, and use the above sample neighborhood distance threshold ∈ to obtain the neighborhood subsample set of the current core point to be determined (that is, obtain a subsample set of the sample set D, this sample set is based on taking the current core point to be determined as the center of the circle, obtain all valid feedback coordinate points in the unvisited sample set P to the current core point to be determined whose distance is less than or equal to the sample neighborhood distance threshold ∈, and determine these valid feedback coordinate points as a set, that is, the neighborhood subsample set of the current core point to be determined); if the number of samples of the neighborhood subsample set of the current core point to be determined is greater than or equal to the minimum threshold MinPts of the number of samples in the sample neighborhood distance threshold ∈, it proves that the density of the neighborhood subsample set of the current core point to be determined is large enough, and the neighborhood subsample set of the current core point to be determined can be completely used as a cluster, which further proves that the current core point to be determined can be used as a core point, and accordingly, the neighborhood subsample set of the current core point to be determined can be determined as the neighborhood subsample set of the core point;
[0145] A new set N is established, and the current core point and the neighborhood subsample set of the current core point are input into the set N as a sample cluster, namely, the cluster of points to be selected; then all samples in the cluster of points to be selected (all samples are valid feedback coordinate points represented by the current core point and all valid feedback coordinate points in the neighborhood subsample set of the current core point) are screened out from the unvisited sample set P, and new core points to be determined are re-determined to form a new cluster of points to be selected, until the number of valid feedback coordinate points in the unvisited sample set P is 0.
[0146] In summary, the example of the present invention proposes an AR glasses display terminal device key input synchronization response control system and method, which synchronizes the screen of the terminal device to the AR glasses through a synchronization module to obtain the AR glasses display screen;
[0147] A virtual button corresponding to a real button in the terminal device screen is set in the AR glasses display screen; at the same time, the AR glasses obtain structured light feedback data of the optical module, and obtain feedback coordinate points of the object above the terminal device according to the structured light feedback data; according to a preset time period t and a threshold value a of the shortest residence time period of the preset feedback coordinate point above the terminal device within the time period t, a plurality of valid feedback coordinate points are screened out from the plurality of feedback coordinate points;
[0148] Clustering multiple valid feedback coordinate points by DBSCAN density clustering method to obtain multiple valid feedback coordinate point clusters; based on the preset minimum valid feedback coordinate point number threshold in the valid feedback coordinate point cluster of the fingertip, the valid feedback coordinate point clusters of other interference objects are screened out, and finally the feedback coordinate point cluster with the largest number in the feedback coordinate point cluster is determined as the target point cluster (that is, the valid feedback coordinate point cluster of the finger);
[0149] A valid feedback coordinate point closest to the terminal device is selected from the target point cluster as the target point (that is, the lowest point of the fingertip is the target point); at the same moment when the target point touches the terminal device, the virtual key on the AR glasses corresponding to the real key of the terminal device touched by the target point is displayed as the largest, and the position of the terminal device touched by the target point corresponds to the position on the AR glasses where a pre-set touch point is displayed (that is, the target point touches the current real key, and the touch point appears on the virtual key of the AR glasses corresponding to the current real key); finally, when controlling the game, the virtual key on the AR glasses controlled by the touch point is synchronized with the real key on the terminal device controlled by the target point, thereby performing synchronous control operations.
[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace part or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A synchronous response control system for key input of an AR glasses display terminal device, characterized in that: Including AR glasses, shells, and terminal equipment; The AR glasses are connected to the housing and the terminal device respectively; the housing is mounted on the terminal device; the housing includes an optical module and a housing body; The terminal device includes a screen recording module, a button module, and a synchronization module; The AR glasses include a display module and a virtual button module; The optical module is used to emit light above the terminal device and obtain structured light feedback data above the terminal device through the light; The screen recording module on the above-mentioned terminal device is used to obtain the screen of the terminal device and send the screen of the terminal device to the display module on the AR glasses; The button module on the terminal device is used to obtain the position of the real button in the terminal device screen, and send the position of the real button in the terminal device screen to the virtual button module of the AR glasses; The virtual key module on the AR glasses is used to set a virtual key at a corresponding position of the AR glasses according to the position of the real key in the terminal device screen sent by the key module; The display module on the AR glasses is used to obtain the screen of the terminal device sent by the screen recording module, and synchronize the screen of the terminal device to the AR glasses; The synchronization module on the terminal device is used to obtain the structured light feedback data in the optical module, and obtain the feedback coordinate point of the fingertip according to the structured light feedback data, and when the vertical distance from the feedback coordinate point of the fingertip to the terminal device is less than the preset shortest distance threshold s between the lowest point of the fingertip and the terminal device, a semi-transparent virtual button is synchronously displayed on the AR glasses; The synchronization module on the above-mentioned terminal device is also used to pre-set touch points on virtual keys on the AR glasses corresponding to real keys of the terminal device; and control the touch points to operate synchronously with the real keys of the terminal device.
2. The AR glasses display terminal device key input synchronous response control system according to claim 1, characterized in that: The synchronization module on the above-mentioned terminal device is specifically used to preset the shortest distance threshold s between the lowest point of the fingertip and the terminal device, obtain the coordinate point of the current lowest point of the fingertip through the structured light feedback data, determine the coordinate point of the current lowest point of the fingertip as the target point; and obtain the vertical distance from the target point to the terminal device; Determine whether the vertical distance from the target point to the terminal device is less than or equal to a threshold s of the shortest distance between the lowest point of the fingertip and the terminal device; If the vertical distance from the target point to the terminal device is less than or equal to the shortest distance threshold s between the lowest point of the fingertip and the terminal device, further determine whether the area below the current target point is located at the location of the area of each real button in the screen of the terminal device; If the area below the current target point is located at the position of the area of each real button in the screen of the terminal device, then the real button below the current target point is determined to be the target real button; and displaying the virtual key corresponding to the target real key in the display screen of the AR glasses as a semi-transparent virtual key; If the vertical distance from the target point to the terminal device is 0, the touch point is displayed at a position corresponding to the position of the target point in the display screen of the AR glasses, the touch point is synchronized with the position of the target point, and corresponding synchronization control operations are performed.
3. The AR glasses display terminal device key input synchronous response control system according to claim 2, characterized in that: The synchronization module is specifically used to obtain a plurality of feedback coordinate points through the structured light feedback data, and obtain a plurality of valid feedback coordinate points from the plurality of feedback coordinate points by using a preset time period t and a preset shortest residence time period threshold a of the feedback coordinate point above the terminal device within the time period t; Using the DBSCAN density clustering method to cluster the multiple valid feedback coordinate points, obtain multiple valid feedback coordinate point clusters, and determine the valid feedback coordinate point clusters as the point clusters to be selected; A minimum threshold value L of the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip is preset, and if the number of valid feedback coordinate points in the to-be-selected point cluster is greater than or equal to the minimum threshold value L of the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip, the to-be-selected point cluster is determined to be a to-be-selected target point cluster; and the to-be-selected target point cluster with the largest number of valid feedback coordinate points in the to-be-selected target point cluster is selected as the target point cluster; The vertical distances from all the valid feedback coordinate points to the terminal device are calculated respectively; and the valid feedback coordinate point in the target point cluster that is the shortest in vertical distance to the terminal device is determined as the target point.
4. A method for controlling the synchronous response of key input of an AR glasses display terminal device, characterized in that: The steps are as follows: The AR glasses establish a connection with the terminal device, and synchronize the screen of the terminal device to the AR glasses to obtain the AR glasses display screen; the AR glasses establish a connection with the optical module, and obtain the structured light feedback data of the optical module in real time; The AR glasses obtain the positions of the regions of each real key in the screen of the terminal device, and set virtual keys at positions corresponding to the positions of the regions of each real key in the screen of the terminal device in the screen of the AR glasses; and preset touch points in the regions where each virtual key is located; The optical module is initialized, a shortest distance threshold s between the lowest point of the fingertip and the terminal device is preset, the coordinate point of the current lowest point of the fingertip is obtained through the structured light feedback data, and the coordinate point of the current lowest point of the fingertip is determined as the target point; and the vertical distance from the target point to the terminal device is obtained; Determine whether the vertical distance from the target point to the terminal device is less than or equal to a threshold s of the shortest distance between the lowest point of the fingertip and the terminal device; If the vertical distance from the target point to the terminal device is less than or equal to the shortest distance threshold s between the lowest point of the fingertip and the terminal device, further determine whether the area below the current target point is located at the location of the area of each real button in the screen of the terminal device; If the area below the current target point is located at the position of the area of each real button in the screen of the terminal device, then the real button below the current target point is determined to be the target real button; and displaying the virtual key corresponding to the target real key in the display screen of the AR glasses as a semi-transparent virtual key; If the vertical distance from the target point to the terminal device is 0, the touch point is displayed at a position corresponding to the position of the target point in the display screen of the AR glasses, the touch point is synchronized with the position of the target point, and corresponding synchronization control operations are performed.
5. The method for controlling the synchronous response of key input of an AR glasses display terminal device according to claim 4, characterized in that: Obtaining the coordinate point of the lowest point of the current fingertip through the structured light feedback data, and determining the coordinate point of the lowest point of the current fingertip as the target point, includes the following steps: Acquire a plurality of feedback coordinate points through the structured light feedback data, and acquire a plurality of valid feedback coordinate points from the plurality of feedback coordinate points; Using the DBSCAN density clustering method to cluster the multiple valid feedback coordinate points, obtain multiple valid feedback coordinate point clusters, and determine the valid feedback coordinate point clusters as the point clusters to be selected; A minimum threshold value L of the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip is preset, and if the number of valid feedback coordinate points in the to-be-selected point cluster is greater than or equal to the minimum threshold value L of the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip, the to-be-selected point cluster is determined to be a to-be-selected target point cluster; and the to-be-selected target point cluster with the largest number of valid feedback coordinate points in the to-be-selected target point cluster is selected as the target point cluster; The vertical distances from all the valid feedback coordinate points to the terminal device are calculated respectively; and the valid feedback coordinate point in the target point cluster that is the shortest in vertical distance to the terminal device is determined as the target point.
6. The method for controlling the synchronous response of key input of an AR glasses display terminal device according to claim 4, characterized in that: The coordinate point of the lowest point of the current fingertip is obtained by using the structured light feedback data, and the coordinate point of the lowest point of the current fingertip is determined as the target point, and the following operation steps are also included: Acquire multiple feedback coordinate points through the structured light feedback data, and acquire multiple valid feedback coordinate points from the multiple feedback coordinate points; perform clustering processing on the multiple valid feedback coordinate points using the DBSCAN density clustering method to acquire multiple valid feedback coordinate point clusters, and determine the valid feedback coordinate point clusters as the point clusters to be selected; A minimum threshold value L of the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip is preset, and the number L' of valid feedback coordinate points in all the candidate point clusters is obtained. If the number L' of valid feedback coordinate points in the candidate point cluster is greater than or equal to the minimum threshold value L of the number of valid feedback coordinate points in the valid feedback coordinate point cluster of the fingertip, the candidate point cluster is determined to be a candidate target point cluster; and the candidate target point cluster with the largest number of valid feedback coordinate points in the candidate target point cluster is selected as the target point cluster; Acquire all valid feedback coordinate points in the target point cluster, preset a maximum 3D distance threshold y between two adjacent valid feedback coordinate points, and calculate the 3D distance between two adjacent valid feedback coordinate points; determine the magnitude relationship between the 3D distance between the two adjacent valid feedback coordinate points and the maximum 3D distance threshold y between the two adjacent valid feedback coordinate points, and if the 3D distance between the two adjacent valid feedback coordinate points is greater than the maximum 3D distance threshold between the two adjacent valid feedback coordinate points, acquire all valid feedback coordinate points whose 3D distance between two adjacent valid feedback coordinate points is greater than the maximum 3D distance threshold between the two adjacent valid feedback coordinate points, and then acquire the 3D distance between other two adjacent valid feedback coordinate points, and calculate the average value of the 3D distance between the other two adjacent valid feedback coordinate points; According to the average value of the 3D distances between the other two adjacent valid feedback coordinate points, constructing the valid feedback coordinate points of the blocked fingertip by simulation, and obtaining a complete cluster of valid feedback coordinate points of the fingertip; The vertical distances from the effective feedback coordinate points to the terminal device are calculated respectively; and the effective feedback coordinate point in the complete effective feedback coordinate point cluster of the fingertip that is closest to the terminal device in vertical distance is determined as the target point.
7. The method for controlling the synchronous response of key input of an AR glasses display terminal device according to claim 5 or claim 6, characterized in that: Acquiring a plurality of feedback coordinate points through the structured light feedback data, and acquiring a plurality of valid feedback coordinate points from the plurality of feedback coordinate points, comprises the following steps: Randomly select any one of the multiple feedback coordinate points as the first feedback coordinate point; preset a time period t, and preset a shortest residence time period threshold a of the feedback coordinate point above the terminal device within the time period t, and obtain a time a' that the current first feedback coordinate point stays above the terminal device within the time period t; if the time a' that the current first feedback coordinate point stays above the terminal device within the time period t is greater than the shortest residence time period threshold a of the feedback coordinate point above the terminal device within the time period t, determine that the current first feedback coordinate point is a valid feedback coordinate point.
8. The method for controlling the synchronous response of key input of an AR glasses display terminal device according to claim 7, characterized in that: The DBSCAN density clustering method is used to cluster the multiple valid feedback coordinate points to obtain multiple valid feedback coordinate point clusters, and the valid feedback coordinate point clusters are determined as the point clusters to be selected, including the following steps: Establish a sample set D, and input the multiple valid feedback coordinate points into the sample set D; preset a sample neighborhood distance threshold ϵ; preset a minimum threshold MinPts of the number of samples in the sample neighborhood distance threshold ϵ; Initialize the unvisited sample set P; Randomly select any feedback coordinate point in the unvisited sample set P as the core point to be determined; According to the sample neighborhood distance threshold ϵ, a neighborhood subsample set of the core point to be determined is obtained, and the number of samples in the neighborhood subsample set of the core point to be determined is obtained; If the number of samples in the neighborhood subsample set of the core point to be determined is greater than or equal to the minimum threshold value MinPts of the number of samples in the sample neighborhood distance threshold ϵ, the core point to be determined is determined to be a core point, and the neighborhood subsample set of the core point to be determined is determined to be a neighborhood subsample set of the core point; Establish a subsample set N, and put the core point and the neighborhood subsample set of the core point into the subsample set N, and determine the set subsample N as the point cluster N to be selected; A new unvisited sample set PN is obtained, and the above operation is repeated until the unvisited sample set P=0, and a plurality of point clusters to be selected are obtained.
9. The method for controlling the synchronous response of key input of an AR glasses display terminal device according to claim 8, characterized in that: The ϵ is a constant; the MinPts is a constant.
