A gesture recognition-based interactive method and system for AR glasses for navigation.

Abstract

This invention relates to the field of augmented reality technology, and more particularly to a gesture recognition-based navigation AR glasses interaction method and system. The invention acquires control commands triggered when the user's gaze is on various page areas within a historical period, analyzes the trigger frequency of these commands for each page area, filters the control commands corresponding to each page area, constructs a set of command triggering patterns specific to each page area, collects user images, identifies age groups, and determines a dynamic coverage coefficient; it acquires the user's gaze in real time, marks the corresponding page areas, and, based on the marked page areas, obtains the required control commands from the corresponding command triggering pattern set; based on the coverage coefficient, it determines the coverage area and generates optimized gesture action samples; and it prioritizes the execution of judgments based on these optimized gesture action samples. This invention adaptively optimizes the judgment criteria based on the different functions of different areas of the page, improving the accuracy of gesture judgment and recognition, accelerating response speed, and reducing the false judgment rate.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and in particular to a gesture recognition-based navigation AR glasses interaction method and system. Background Technology

[0002] AR devices are often used as wearable terminals for guiding tours in museums, exhibition halls, and industrial sites. They are exposed to complex interactive environments with multiple users, multiple lighting conditions, and multiple occlusions for a long time. Due to differences in age, gesture habits, and scene page functions, gesture recognition of AR devices is prone to false touches or missed detections. To solve these problems, technologies such as convolutional neural networks have been introduced to improve gesture recognition accuracy.

[0003] For example, Chinese Patent Publication No. CN108334814A discloses an AR system gesture recognition method based on convolutional neural networks combined with user habitual behavior analysis, including the following steps: Step 1: User habitual gesture image acquisition: A set of gestures is randomly provided by the user, and this set of gestures is used as the standard gesture. The images of this set of gestures are acquired and recorded as the standard group; the corresponding actual label category is constructed according to the standard group gesture model diagram; different label categories are set to trigger the corresponding AR system specified functions; Step 2: Gesture region image detection: Gesture region image detection is performed on the gesture images of the standard group, training sample group and test sample group respectively to achieve the segmentation of skin-colored and non-skin-colored regions in the image; Step 3: Convolutional neural network to realize gesture feature recognition: A preliminary structure model of convolutional neural network is designed, the convolutional neural network model is trained and tested with sample data, and the binarized image is directly input into the convolutional neural network.

[0004] However, the following problems still exist in the existing technology:

[0005] In existing technologies, due to functional differences in different areas of an AR page, the frequency of triggering certain commands varies. Existing technologies do not consider the impact of functional differences in different areas of the page on command triggering, and use the same motion capture and judgment methods, resulting in poor capture of gestures and inaccurate results. Summary of the Invention

[0006] To address this, the present invention provides a gesture recognition-based AR glasses interaction method and system to overcome the problems in the prior art where the frequency of triggering some commands varies due to functional differences in different areas of the AR page, and the prior art does not consider the impact of functional differences in different areas of the page on command triggering, and uses the same motion capture and judgment method, resulting in poor capture of gestures and inaccurate results.

[0007] To achieve the above objectives, in one aspect, the present invention provides a gesture recognition-based interactive method for AR glasses, comprising:

[0008] Get the control commands triggered when the user's eye gaze is in each page area during the historical period, and extract the gesture trajectory corresponding to the triggering control commands;

[0009] Based on the triggering frequency of control commands in each page area, control commands are filtered to construct a set of command triggering patterns for each page area;

[0010] In response to acquiring a user's image, the user's age group is determined, and a dynamic inclusion coefficient is determined based on the age group;

[0011] In response to obtaining the user's eye gaze point, the page area where the eye gaze point is located is determined. Based on the set of instruction triggering rules corresponding to the page area, the required control instructions and dynamic tolerance coefficient are selected to optimize the determination of the gesture trajectory corresponding to the control instructions.

[0012] Determine the gesture action sample corresponding to each of the selected control commands, and optimize the gesture action sample, including constructing a coverage area covering the trajectory based on the trajectory in the gesture action sample to form an optimized gesture action sample;

[0013] Collect the user's gesture trajectory and match it with gesture samples and optimized gesture samples to determine whether the gesture matches and to determine the triggered control command.

[0014] The gesture trajectory includes the trajectory corresponding to the recognition point.

[0015] Furthermore, based on the triggering frequency of control commands in each page area, the control commands are filtered, including...

[0016] Calculate the historical trigger frequency values ​​of various control commands in the corresponding page areas;

[0017] Determine the function type corresponding to each page area, and set a frequency threshold benchmark value based on the function type of the page area;

[0018] Filter control commands whose historical trigger frequency values ​​are greater than or equal to the corresponding area frequency threshold reference value;

[0019] The page area has two functional types: a core operation area and an auxiliary operation area. The core operation area uses a first frequency threshold reference value, and the auxiliary operation area uses a second frequency threshold reference value.

[0020] Furthermore, a set of instruction triggering rules for each page region is constructed, including:

[0021] Create a storage collection for each page area;

[0022] Store the selected and retained control commands into the storage collection of the corresponding page area;

[0023] The storage set of the page area is determined as the set of instruction triggering rules.

[0024] Furthermore, the user's age group is determined, and a dynamic inclusion coefficient is established based on this age group, including:

[0025] Determine the user's age group based on the user's image;

[0026] The dynamic inclusion coefficient is determined based on the user's age group;

[0027] The correlation between age group and dynamic tolerance coefficient value is predetermined.

[0028] Furthermore, the page area where the eye's gaze point is located is determined, and the required control instructions are selected based on the set of instruction triggering rules corresponding to the page area, including:

[0029] Obtain the real-time coordinates of the eye's gaze point on the page to determine the gaze range;

[0030] The gaze range is matched with the predefined coordinate range of each page region, and the corresponding page region is marked.

[0031] Select the various control instructions from the instruction triggering rule set corresponding to the marked page area as the demand control instructions.

[0032] Furthermore, based on the trajectory in the gesture action sample, a coverage area covering the trajectory is constructed to form an optimized gesture action sample, including,

[0033] The outline in the gesture action sample is determined, and virtual reference lines parallel to the outline are constructed on both sides of the outline to form a strip-shaped coverage area covering the trajectory.

[0034] The covered area is identified as an optimized gesture sample;

[0035] The width of the coverage area is determined based on a dynamic containment coefficient.

[0036] Furthermore, the user's gesture trajectory is collected and matched with gesture samples, including...

[0037] Determine the similarity between the gesture trajectory and the gesture sample. If the similarity is greater than a predetermined similarity threshold, then the gesture trajectory is determined to match the gesture sample.

[0038] Furthermore, the user's gesture trajectory is collected and matched with optimized gesture samples, including...

[0039] The real-time collected gesture trajectory is subjected to spatial transformation processing and coverage detection is performed with each optimized gesture sample;

[0040] If at least one spatially transformed gesture trajectory falls completely within the coverage area corresponding to the optimized gesture sample, then the gesture trajectory is determined to match the optimized gesture sample.

[0041] The spatial transformation process includes translation transformation and rotation transformation.

[0042] Furthermore, the gesture action samples are preferentially matched with each optimized gesture action sample.

[0043] On the other hand, the present invention also provides a system for applying a gesture recognition-based interactive method for guided AR glasses, comprising,

[0044] The data acquisition module is connected to the AR glasses sensor and the AR interaction system sensor array to acquire user eye gaze coordinate data, real-time gesture trajectory data and user image data.

[0045] The storage management module, which is connected to the data acquisition module, is used to store the set of instruction triggering rules, gesture action samples, and page area coordinate data;

[0046] The instruction optimization and pattern construction module, which is connected to the storage management module, is used to construct a set of instruction triggering patterns and generate optimized gesture action samples;

[0047] The age determination module, which is connected to the data acquisition module, is used to determine the user's age group based on the user's image and determine the corresponding dynamic tolerance coefficient.

[0048] The real-time interactive control module, which is connected to the instruction optimization and pattern construction module, is used to obtain the user's eye gaze point, determine the page area where the eye gaze point is located, and select the required control instruction based on the instruction trigger pattern set corresponding to the page area.

[0049] A sample generation module, which is connected to the real-time interactive control module, is used to construct a coverage area covering the trajectory based on the trajectory in the gesture action sample, thereby forming an optimized gesture action sample;

[0050] The judgment and execution module is connected to the sample generation module. It is used to match the user's real-time gesture trajectory with the gesture sample and the optimized gesture sample to determine whether the gesture matches and to determine the control command triggered.

[0051] Compared with existing technologies, this invention obtains control commands triggered when the user's eye gaze is in each page area within a historical period, analyzes the triggering frequency of control commands in each page area, filters control commands corresponding to page areas, constructs a set of command triggering patterns specific to each page area, collects user images, identifies age groups, and determines dynamic coverage coefficients; it acquires eye gaze points in real time, marks the corresponding page areas, and obtains the required control commands from the corresponding command triggering pattern set based on the marked page areas, determines the coverage area based on the coverage coefficient, and generates optimized gesture action samples; it prioritizes the execution of judgments based on the optimized gesture action samples. This invention adaptively optimizes the judgment criteria based on the different functions of different areas of the page, improving the accuracy of gesture judgment and recognition, accelerating response speed, and reducing the false judgment rate.

[0052] In particular, this invention constructs a set of instruction triggering rules for each page area, considering various control instructions that are frequently triggered in different areas. In reality, different page areas correspond to different interactive components and functional configurations. Therefore, when a user looks at different areas, the control instructions triggered in the corresponding page areas also differ. Based on this, this invention considers constructing a set of instruction triggering rules to characterize the more frequent control instructions in different page areas, so as to facilitate the subsequent use of different gesture matching methods, thereby making it easier to capture frequent control instructions, reduce the probability of accidental touches of low-frequency triggered control instructions, and improve the matching efficiency and accuracy of gesture actions.

[0053] In particular, this invention considers using different gesture matching methods for different commands. The gesture samples corresponding to control commands in the command triggering pattern set are optimized to generate optimized gesture samples with a dynamic, band-like encompassing region, and these optimized samples are prioritized for matching. This improves the efficiency and accuracy of gesture matching.

[0054] In particular, this invention takes into account the differences in user age groups. In reality, children's and elderly people's gestures may be non-standard or differ significantly from standard gestures, making them more likely to be missed. This invention determines the age group through user images and outputs a dynamic coverage coefficient. Based on this coefficient, the trajectory set area is magnified to generate optimized gesture action samples, achieving adaptive adjustments for specific age groups. This improves the recognition success rate for specific age groups and reduces the false touch rate and false detection rate of gesture judgment.

[0055] In particular, this invention considers spatial transformation processing of the real-time collected gesture trajectory when performing coverage detection with various optimized gesture action samples. During actual interaction, user position shifts or wrist rotations can easily cause trajectory deviations, making mismatches highly likely if only the original trajectory is compared. This invention first performs translational and rotational transformations on the real-time gesture trajectory within the judgment execution module, and then performs matching judgment with the optimized gesture action samples; as long as the spatially transformed trajectory completely falls within the coverage area, it is considered a successful match with the optimized gesture action sample, improving the error tolerance of the recognition standard and increasing the accuracy of gesture judgment. Attached Figure Description

[0056] Figure 1 This is a schematic diagram illustrating the steps of an embodiment of the invention;

[0057] Figure 2 A schematic diagram illustrating the filtering process when the number of reserved control instructions in an embodiment of the invention is zero;

[0058] Figure 3 This is a schematic diagram of the coverage area of ​​an embodiment of the invention;

[0059] Figure 4 This is a logic block diagram illustrating the priority matching of gesture trajectory with optimized gesture samples in an embodiment of the invention.

[0060] Figure 5 This is a schematic diagram of the module connections of the system according to an embodiment of the invention;

[0061] Wherein, 1: the outline of the gesture sample, 2: the coverage area, 3: the virtual reference line, and 4: the width of the coverage area. Detailed Implementation

[0062] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0063] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0064] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0065] Please see Figure 1 As shown, this is a schematic diagram of the steps in an embodiment of the present invention. The gesture recognition-based AR glasses interaction method of the present invention includes:

[0066] Step S1: Obtain the control commands triggered when the user's eye gaze point is in each page area within the historical period, and extract the gesture trajectory corresponding to the triggering control commands;

[0067] Step S2: Based on the triggering frequency of control commands in each page area, filter the control commands and construct a set of command triggering patterns for each page area;

[0068] Step S3: In response to acquiring the user image, determine the user's age group and determine the dynamic coverage coefficient based on the age group;

[0069] Step S4: In response to obtaining the user's eye gaze point, determine the page area where the eye gaze point is located, and select the required control command and dynamic containment coefficient based on the set of command triggering rules corresponding to the page area to optimize the determination of the gesture trajectory corresponding to the control command.

[0070] Step S5: Determine the gesture action sample corresponding to each of the selected control commands, and optimize the gesture action sample, including constructing a coverage area covering the trajectory based on the trajectory in the gesture action sample to form an optimized gesture action sample;

[0071] Step S6: Collect the user's gesture trajectory and match it with the gesture sample and optimized gesture sample to determine whether the gesture matches and to determine the triggered control command.

[0072] The gesture trajectory includes the trajectory corresponding to the recognition point.

[0073] Specifically, the page area refers to a local area corresponding to the AR interactive interface. Those skilled in the art can divide it according to the actual situation, which will not be elaborated here.

[0074] Specifically, control commands include various types, such as click, move, close, zoom, etc., depending on the configuration of interactive components in the local area. In practice, a correspondence between gesture motion trajectories and control commands is pre-built. Once the gesture motion trajectory is recognized or matched, the corresponding control command can be determined and triggered to achieve the corresponding function. This will not be elaborated further.

[0075] Specifically, extracting the gesture trajectory corresponding to the control command, including,

[0076] Record the trajectory of the gesture action corresponding to the triggered control command.

[0077] Specifically, when acquiring the gesture trajectory, the spatial coordinates of the recognition points during the gesture can be recorded in chronological order to form the gesture trajectory.

[0078] Understandably, recognition points can be key skeletal joints located based on gesture recognition technology, such as the fingertips, knuckles, the center of the palm, and spatial locations of the wrist.

[0079] Specifically, there are no restrictions on the method for determining the spatial coordinates of the recognition point. Typically, a binocular camera device capable of acquiring three-dimensional information can be set on the AR glasses to capture the spatial coordinates of a single recognition point, thereby forming the gesture trajectory. This will not be elaborated further.

[0080] Specifically, no particular limitations are made on the technical means of obtaining the user's eye gaze point. It is understood that AR glasses or other AR wearable devices have the function of confirming the eye gaze point, and those skilled in the art can choose the appropriate technical solution based on different devices, which will not be elaborated here.

[0081] Specifically, control commands are filtered based on their triggering frequency in different page areas, including...

[0082] Calculate the historical trigger frequency values ​​of various control commands in the corresponding page areas;

[0083] Determine the function type corresponding to each page area, and set a frequency threshold benchmark value based on the function type of the page area;

[0084] Filter control commands whose historical trigger frequency values ​​are greater than or equal to the corresponding area frequency threshold reference value;

[0085] The page area has two functional types: a core operation area and an auxiliary operation area. The core operation area uses a first frequency threshold reference value, and the auxiliary operation area uses a second frequency threshold reference value.

[0086] Specifically, if the number of control commands whose historical trigger frequency value is greater than or equal to the corresponding regional frequency threshold benchmark value is 0, then the control command with the highest trigger frequency value in that region is added as the screening result.

[0087] Please see Figure 2 As shown, Figure 2 A schematic diagram illustrating the filtering process when the number of control commands to be retained is zero.

[0088] Specifically, no specific criteria are specified for dividing the core operation area and the auxiliary operation area. For example, the core operation area is usually the close and minimize operation area in the upper right corner of the page, as well as the area in the center of the page where interactive components are relatively concentrated. Essentially, it is a certain high-frequency click area in the interaction. The average historical trigger frequency value of each page area can be calculated. To reflect the high-frequency click situation, if the historical trigger frequency value of a page area is greater than 1.3 times the average historical trigger frequency value, then the page area is determined to be the core operation area.

[0089] Non-core operation areas are designated as auxiliary operation areas.

[0090] Specifically, the first frequency threshold benchmark value is determined based on the historical operation data of the core operating area. The historical trigger frequency values ​​of the three types of control commands with the highest trigger frequency in the historical period of each core operating area are statistically analyzed, and the arithmetic mean of the historical trigger frequency values ​​of these three types of control commands is calculated. This arithmetic mean is set as the first frequency threshold benchmark value of the corresponding core operating area.

[0091] Specifically, the second frequency threshold benchmark value is determined based on the historical operation data of the auxiliary operation area. The historical trigger frequency values ​​of the three types of control commands with the highest trigger frequency in the historical period of each auxiliary operation area are statistically analyzed, and the arithmetic mean of the historical trigger frequency values ​​of these three types of control commands is calculated. This arithmetic mean is set as the second frequency threshold benchmark value of the corresponding auxiliary operation area.

[0092] Specifically, this involves constructing a set of instruction triggering rules for each page region, including...

[0093] Create a storage collection for each page area;

[0094] Store the selected and retained control commands into the storage collection of the corresponding page area;

[0095] The storage set of the page area is determined as the set of instruction triggering rules.

[0096] Specifically, it involves determining the user's age group and then setting a dynamic inclusion factor based on that age group, including...

[0097] Determine the user's age group based on the user's image;

[0098] The dynamic inclusion coefficient is determined based on the user's age group;

[0099] The correlation between age group and dynamic tolerance coefficient value is predetermined.

[0100] Specifically, the method uses an open-source image age recognition model to identify user images and determine the user's age group. The specific form of the image age recognition model is not limited; those skilled in the art can choose an open-source technical solution based on the actual situation, which will not be elaborated upon here.

[0101] Specifically, the age groups are pre-defined as follows: 0-5 years old (preschool), 6-12 years old (children), 13-45 years old (adolescents and adults), and over 45 years old (middle-aged and elderly).

[0102] Specifically, the dynamic inclusion coefficient is 1.4 for children aged 0-5 years and older (over 45 years old). The dynamic inclusion coefficient is 1.2 for children aged 6-12 years and older (over 45 years old). The dynamic inclusion coefficient is 1.05 for adolescents and young adults aged 13-45 years.

[0103] Understandably, for toddlers aged 0-5, their hand gestures are very small and their positioning accuracy is poor during their early development. In addition, toddlers lack understanding of AR, so a larger dynamic tolerance coefficient is chosen. For teenagers and adults aged 13-45, their hand gestures are stable and they usually have some understanding of AR, so a smaller dynamic tolerance coefficient is chosen.

[0104] The correlation between age group and dynamic tolerance coefficient value is predetermined.

[0105] Specifically, the page area where the eye's gaze point is located is determined, and the required control instructions are selected based on the set of instruction triggering rules corresponding to that page area, including...

[0106] Obtain the real-time coordinates of the eye's gaze point on the page to determine the gaze range;

[0107] The gaze range is matched with the predefined coordinate range of each page region, and the corresponding page region is marked. If there is an overlap between the page region and the gaze range, the page region is marked.

[0108] Select the various control instructions from the instruction triggering rule set corresponding to the marked page area as the demand control instructions.

[0109] Specifically, the gaze range is a circular area constructed with the real-time coordinates of the eye's gaze point on the page as the center and a predetermined radius. The size of the predetermined radius depends on the visual observation range at the corresponding viewing distance and can be set to half of the maximum width corresponding to the maximum visual observation range.

[0110] It is understandable that the eye's fixation point reflects the range of the user's visual attention. In reality, a person's line of sight is a variable range rather than a geometrically defined one-dimensional point; therefore, a fixation range should be defined to replace the point concept.

[0111] Understandably, when the eye's gaze point is near the boundary of different areas on the page, multiple page areas should be marked using the gaze range, rather than mechanically considering only the area where the eye's gaze point is located. This would reduce the probability of system misjudgment in some cases, such as when users hesitate or compare browsing guide content at the same time.

[0112] Specifically, please refer to Figure 3 As shown, Figure 3 A schematic diagram of the coverage area in an embodiment of the present invention. A coverage area covering the trajectory in the gesture action sample is constructed to form an optimized gesture action sample, including:

[0113] The outline in the gesture action sample is determined, and virtual reference lines parallel to the outline are constructed on both sides of the outline to form a strip-shaped coverage area covering the trajectory.

[0114] The covered area is identified as an optimized gesture sample;

[0115] The width of the coverage area is determined based on a dynamic containment coefficient.

[0116] It is understandable that the coverage area is strip-shaped, and the width of the coverage area is the distance between the virtual reference line and the outline;

[0117] In implementation, the width of the coverage area is set as the product of the dynamic containment coefficient and the standard distance;

[0118] The standard distance is preset. Taking into account the range of movement of the gesture, the standard distance is selected within the range of [2cm, 4cm] to allow for a certain lateral offset during the trajectory movement.

[0119] Specifically, the gesture samples are predetermined. During implementation, different gesture samples and control commands are pre-defined to trigger the corresponding control commands when the user makes a corresponding gesture.

[0120] Specifically, the process involves collecting the user's hand gesture trajectories and matching them with hand gesture samples, including...

[0121] Determine the similarity between the gesture trajectory and the gesture sample. If the similarity is greater than a predetermined similarity threshold, then the gesture trajectory is determined to match the gesture sample.

[0122] Specifically, similarity matching is based on similarity algorithms, such as the cosine similarity algorithm. The specific implementation of the algorithm is not limited; those skilled in the art can choose the appropriate similarity algorithm based on actual needs, as long as it can determine the degree of trajectory similarity and output a similarity value. Further details will not be elaborated here.

[0123] Specifically, the similarity threshold is determined based on historical similarity values. A person skilled in the art imitates the predetermined gesture to make the actual gesture, collects several gesture samples, and calculates the mean similarity between the several gesture samples. To allow for a certain error, the similarity threshold is set as the product of the mean similarity and the error coefficient, and the error coefficient is selected in the interval [0.8, 0.9].

[0124] Specifically, the process involves collecting the user's gesture trajectory and matching it with optimized gesture samples, including...

[0125] The real-time collected gesture trajectory is subjected to spatial transformation processing and coverage detection is performed with each optimized gesture sample;

[0126] If at least one spatially transformed gesture trajectory falls completely within the coverage area corresponding to the optimized gesture sample, then the gesture trajectory is determined to match the optimized gesture sample.

[0127] The spatial transformation process includes translation transformation and rotation transformation.

[0128] In practice, the collected user gesture trajectory and optimized gesture samples are normalized, scaled to the same scale, placed in a unified coordinate system for spatial transformation and coverage detection, which will not be elaborated further.

[0129] Specifically, the gesture action samples are preferentially matched with each optimized gesture action sample.

[0130] Specifically, priority is given to matching gesture trajectories with optimized gesture samples. Please refer to [link / reference]. Figure 4 As shown, Figure 4 The logic diagram for prioritizing the matching of gesture motion trajectories with optimized gesture motion samples.

[0131] The spatially transformed gesture trajectory is compared with the coverage area of ​​each optimized gesture sample to determine whether the gesture trajectory matches the optimized gesture sample.

[0132] If no optimized gesture action sample is successfully matched, the gesture action trajectory is matched with each of the gesture action samples based on similarity.

[0133] The coverage detection of optimized gesture action samples adopts a reverse order traversal method based on historical trigger frequency values ​​to select corresponding optimized gesture action samples for matching, while the similarity matching with gesture action samples adopts a random order traversal method.

[0134] On the other hand, this invention also provides a gesture recognition-based AR glasses interactive system for navigation, please refer to... Figure 5 As shown, Figure 5 This is a schematic diagram of the system's module connections. The gesture recognition-based AR glasses interactive system for navigation includes:

[0135] The data acquisition module is connected to the AR glasses sensor and the AR interaction system sensor array to acquire user eye gaze coordinate data, real-time gesture trajectory data and user image data.

[0136] The storage management module, which is connected to the data acquisition module, is used to store the set of instruction triggering rules, gesture action samples, and page area coordinate data;

[0137] The instruction optimization and pattern construction module, which is connected to the storage management module, is used to construct a set of instruction triggering patterns and generate optimized gesture action samples;

[0138] The age determination module, which is connected to the data acquisition module, is used to determine the user's age group based on the user's image and determine the corresponding dynamic tolerance coefficient.

[0139] The real-time interactive control module, which is connected to the instruction optimization and pattern construction module, is used to obtain the user's eye gaze point, determine the page area where the eye gaze point is located, and select the required control instruction based on the instruction trigger pattern set corresponding to the page area.

[0140] A sample generation module, which is connected to the real-time interactive control module, is used to construct a coverage area covering the trajectory based on the trajectory in the gesture action sample, thereby forming an optimized gesture action sample;

[0141] The judgment and execution module is connected to the sample generation module. It is used to match the user's real-time gesture trajectory with the gesture sample and the optimized gesture sample to determine whether the gesture matches and to determine the control command triggered.

[0142] Specifically, there are no restrictions on the form of the storage management module; it can be a data storage device capable of data interaction and storage, which will not be elaborated further.

[0143] Specifically, there are no restrictions on the specific forms of the data acquisition module, instruction optimization and pattern construction module, real-time interactive control module, sample generation module, and judgment execution module. They can be composed of logic components, including field-programmable processors, computers, or microprocessors in computers, which will not be elaborated further.

[0144] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A gesture recognition-based interactive method for AR glasses, characterized in that: Get the control commands triggered when the user's eye gaze is in each page area during the historical period, and extract the gesture trajectory corresponding to the triggering control commands; Based on the triggering frequency of control commands in each page area, control commands are filtered to construct a set of command triggering patterns for each page area; In response to acquiring a user's image, the user's age group is determined, and a dynamic inclusion coefficient is determined based on the age group; In response to obtaining the user's eye gaze point, the page area where the eye gaze point is located is determined. Based on the set of instruction triggering rules corresponding to the page area, the required control instructions and dynamic tolerance coefficient are selected to optimize the determination of the gesture trajectory corresponding to the control instructions. Determine the gesture action sample corresponding to each of the selected control commands, and optimize the gesture action sample, including constructing a coverage area covering the trajectory based on the trajectory in the gesture action sample to form an optimized gesture action sample; Collect the user's gesture trajectory and match it with gesture samples and optimized gesture samples to determine whether the gesture matches and to determine the triggered control command. The gesture trajectory includes the trajectory corresponding to the recognition point. The process of filtering control commands based on their trigger frequency in each page area includes... Calculate the historical trigger frequency values ​​of various control commands in the corresponding page areas; Determine the function type corresponding to each page area, and set a frequency threshold benchmark value based on the function type of the page area; Filter control commands whose historical trigger frequency values ​​are greater than or equal to the corresponding area frequency threshold reference value; The functional types of the page area include a core operation area and an auxiliary operation area. The core operation area adopts a first frequency threshold reference value, and the auxiliary operation area adopts a second frequency threshold reference value. The process of constructing a set of instruction triggering rules for each page region includes, Create a storage collection for each page area; Store the selected and retained control commands into the storage collection of the corresponding page area; The storage set of the page area is determined as the set of instruction triggering rules; The process of determining a user's age group and then setting a dynamic inclusion factor based on that age group includes: Determine the user's age group based on the user's image; The dynamic inclusion coefficient is determined based on the user's age group; The correlation between age groups and dynamic tolerance coefficient values ​​is predetermined; The process of obtaining the real-time coordinates of the eye's gaze point on the page to determine the gaze range includes, The gaze range is matched with the predefined coordinate range of each page region, and the corresponding page region is marked. Select the various control instructions from the instruction triggering rule set corresponding to the marked page area as the demand control instructions; The process of constructing a coverage area that covers the trajectory in the gesture action sample to form an optimized gesture action sample includes: The outline in the gesture action sample is determined, and virtual reference lines parallel to the outline are constructed on both sides of the outline to form a strip-shaped coverage area covering the trajectory. The covered area is identified as an optimized gesture sample; The width of the coverage area is determined based on a dynamic containment coefficient.

2. The gesture recognition-based AR glasses interaction method according to claim 1, characterized in that, Collect the user's gesture trajectory and match it with gesture samples, including... Determine the similarity between the gesture trajectory and the gesture sample. If the similarity is greater than a predetermined similarity threshold, then the gesture trajectory is determined to match the gesture sample.

3. The gesture recognition-based AR glasses interaction method according to claim 1, characterized in that, Collect user gesture trajectory data and match it with optimized gesture samples, including: The real-time collected gesture trajectory is subjected to spatial transformation processing and covered by each optimized gesture sample. If at least one spatially transformed gesture trajectory falls completely within the coverage area corresponding to the optimized gesture sample, then the gesture trajectory is determined to match the optimized gesture sample. The spatial transformation process includes translation transformation and rotation transformation.

4. The gesture recognition-based AR glasses interaction method according to claim 3, characterized in that, It also includes prioritizing the matching of the gesture action samples with each optimized gesture action sample.

5. A system applying the gesture recognition-based AR glasses interaction method according to any one of claims 1-4, characterized in that, include, The data acquisition module is connected to the AR glasses sensor and the AR interaction system sensor array to acquire user eye gaze coordinate data, real-time gesture trajectory data and user image data. The storage management module, which is connected to the data acquisition module, is used to store the set of instruction triggering rules, gesture action samples, and page area coordinate data; The instruction optimization and pattern construction module, which is connected to the storage management module, is used to construct a set of instruction triggering patterns and generate optimized gesture action samples; The age determination module, which is connected to the data acquisition module, is used to determine the user's age group based on the user's image and determine the corresponding dynamic tolerance coefficient. The real-time interactive control module, which is connected to the instruction optimization and pattern construction module, is used to obtain the user's eye gaze point, determine the page area where the eye gaze point is located, and select the required control instruction based on the instruction trigger pattern set corresponding to the page area. A sample generation module, which is connected to the real-time interactive control module, is used to construct a coverage area covering the trajectory based on the trajectory in the gesture action sample, thereby forming an optimized gesture action sample; The judgment and execution module is connected to the sample generation module. It is used to match the user's real-time gesture trajectory with the gesture sample and the optimized gesture sample to determine whether the gesture matches and to determine the control command triggered.

Images